Just wondering what kind of things do you do in your industry.
3 Likes
I used to work at a large multinational company and used Julia to analyze CAN bus together with other log files. These files are huge, so you need fast code that can easily be adapted to changing requirements.
7 Likes
Interesting, did you find good packages for industrial buses in julia ?
Not so many. There is for example: GitHub - tsabelmann/CANalyze.jl: Julia package for analyzing CAN-bus data using messages and variables.
And GitHub - analytech-solutions/Modbus.jl .
More for robotics: https://juliarobotics.org/
6 Likes
I do something similar with Reddit - Dive into anything .
3 Likes
I am doing real time signal processing, i.e. processing sensor data and making sense of it. But there is also quite some prototyping and tooling involved. We need to interact with existing systems, so interoperability is a reoccurring topic.
9 Likes
I am not sure if this is considered “work industry” but I am using Julia for working voluntarily with a Fire District Department. Mainly as a analytical tool, data viz and reporting of emergencies in a city (house fire, car accidents, rescue, etc). So I am basically your average dataframes, plots.jl, makie, guy hahaha.
Because we finally could incorporate IT solutions to track emergencies almost one year ago, I am consider to include some dashboard solution (maybe Genie.jl), and from that, start trying predictive and prescriptive models.
18 Likes
I work for PumasAI. I (among other colleagues) work on developing and maintaining proprietary statistical software in Julia with a main focus on hierarchical, nonlinear mixed effects (NLME), ODE-based models. We also combine machine learning with NLME models in various ways for covariate and dynamical model identification given sparse population data.
8 Likes
Bioinformatics, sequencing data & statistics/modeling in that domain.
2 Likes
We consume a “firehose” of flight events for every active and scheduled flight in the world, keeping both a local in-memory “state” of each flight, as well as backing up to a redis server.
The JSON events come in every 1-3 ms, and we have to balance doing a flight state lookup, possibly from redis, along with a few other lookups, calculations, and joins, and possibly kicking off notifications for delayed/cancelled flights. Using Julia is key to carefully interleaving the operations and keeping allocations to a minimum so we don’t “fall behind” (i.e. the average processing time per event can’t > 1ms).
16 Likes
We mostly do signal processing but also some other analysis, mostly for large industry. We are working on predictive maintenance where we collect machine speed and vibration data to predict failures. The MCC 172 DAQ HAT is an IEPE vibration data collector. The Julia clang package and ccall functionality works nicely to interact with this unit to collect data and we can process it immediately after collection while collecting another batch of data. This package has only been tested on the MCC 172 HAT, it will need work for the other HATS.
The Raspberry PI 4 and 5 seem to be up to the task, except their graphics capability is weak for many plotting packages. The best package for the PI that I have tried is InspectDR.jl but it has not been updated to work on Julia 1.11.
5 Likes
May I ask how you deal with GC pauses?
Or is this primarily a throughput-oriented job and you don’t particularly care about occasional latency spikes?
(I could imagine e.g. a setup where you have two servers that replicate of most internal state; at each time only one of them actually produces output and has GC disabled, while the other is permitted to GC and otherwise processes replicated events / catches up buffered events that piled up during the last GC pause)
2 Likes
We haven’t had to worry too much about GC pauses, mainly because we’ve aimed to keep allocations per processed event very low (a few hundred very small allocations). We’ve had a couple cases where a type instability or accidental large allocation was introduced and it ends up hurting the overall throughput, but as long as we stay under a certain threshold (not super well defined in this case), the occasional GC times are minor enough that overall throughput per processed event isn’t largely affected.
I know in some workloads, they could be more GC-sensitive, but having 1-3ms per event to process is actually quite a bit of time, relatively speaking, to make sure we have some buffer time for GC every once in a while. We typically average around 300 microseconds of processing per event, with best cases around 100 and worst cases being the long-tail of several milliseconds (worst-case having to do full network calls + GC, etc.).
6 Likes
So you’re mostly on throughput? Because even very moderate allocations will mean that you get a full GC once a week, which means that you miss a very generous 100ms deadline on an event that should have taken an average/amortized 100us.
This kind of consideration drives a lot of the GC internals of golang, and java G1 / ZGC/ shenandoah; julia only has the analogue of the throughput-oriented java “parallel”. So I was wondering how people deal with that.
(apologies for potentially derailing the very interesting thread – we could split that convo off?)
2 Likes
I work on a wind tunnel and we do a lot of the data acquisition and most data processing in Julia.
6 Likes
I’m a trader (mostly Australian and US equities).
I use Julia for storing data, estimating models, and making predictions. A lot of the statistical routines I’ve written myself, but I do also use Dataframes.jl, StatsBase.jl, Distributions.jl, Combinatorics.jl, Flux.jl, and NLopt.jl. Oh, and Timezones.jl really comes in handy as well.
For interacting with broker APIs, or anything to do with language models, I use Python, and so therefore PyCall.jl gets used a fair bit. I know that PythonCall.jl is the more heavily maintained package now but I’ve never got around to updating my code to work with it.
2 Likes
…crazy that Julia is barely supported by brokers’ APIs so far. Thought IBKR would’ve supported Julia by now.