# State of machine learning in Julia
**URL:** https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385
**Category:** Machine Learning
**Created:** [January 11, 2022, 10:39am UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385 "2022-01-11T10:39:27Z")
**Posts on this page:** 20
**Page:** 3
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [January 14, 2022, 10:54pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/41 "2022-01-14T22:54:07Z")
> [@Thibaut](#):
>
> But it is easier to think that I will use Julia for analysis on a Notebook or some simulation, than a NLP Model with BERT that I may have later to deploy on Mobile.
You might want to track the progress on [ONNX.jl](https://github.com/FluxML/ONNX.jl) - when it’s ready, you should be able to load any of the thousands of the pretrained models to Julia (e.g. see [instructions for HuggingFace models](https://huggingface.co/docs/transformers/serialization)) as well as run Julia models [on mobile](https://onnxruntime.ai/docs/tutorials/mobile/). We are not there yet, but some simple examples like Resnet are already [functional](https://github.com/FluxML/ONNX.jl/blob/master/examples/resnet18.jl).
* * *
One obstacle though, and, in my opinion, the biggest **challenge** for Julia, is that it provides an **open and extensible ecosystem** for machine learning. For example, JAX has ~200 operators, ONNX - ~250, PyTorch - around 2000. Every operator there is guaranteed to support backpropagation, work on GPU/TPU, have good documentation, etc. Sometimes [it’s hard](https://dev-discuss.pytorch.org/t/where-do-the-2000-pytorch-operators-come-from-more-than-you-wanted-to-know/373) to support all of them, yet possible.
In Julia, we essentially have an infinite list of operators scattered over multiple packages. There are no formal requirements for adding new functions, and authors usually only cover their own needs. For instance, someone adding a new `ChainRules.rrule()` may not check if it works with `CuArray`s, somebody creating a new closure-based layer may not think about exporting it to ONNX, etc.
For a couple of days I’ve been thinking of a **curated list of high-quality operators and language features** with particular guarantees, e.g.:
- support autodiff (e.g. via `ChainRules.rrule`)
- support GPU
- support low-precision element types (Float32, Float16)
- have docstrings & be discoverable (e.g. be listed in some popular document)
- have performance tests
- support import/export to/from ONNX, XLA, etc.
We can then create automatic tests to ensure the quality of the listed features. Tests themselves don’t have to be exhaustive - these things must be tested in their packages anyway - instead, they will show the level of maturity of the ecosystem and highlight potential issues.
---
### Author:  [@Akatz](https://discourse.julialang.org/u/Akatz)
#### Post date: [January 15, 2022, 11:43pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/42 "2022-01-15T23:43:33Z")
To me that’s just a reframing of the same problem. If you’re restricted to a smaller set of centralized blessed ops, then you might as well use pytorch.
I ideally there would be a way to typecheck functions or have an API that enforces semantics which are more easily optimizable so that an ecosystem could be both distributed and fast/correct with AD. Right now it’s a bit magical. Type stability can be hard enough, but now there are ever shifting inscrutable code patterns and corner cases that people have to worry about for GPU and AD, so of course they won’t compose well with flux.
The easier it is technically, the less social coordination has to happen.
BTW, ONNX.jl is amazing. Whatever happens in Julia, it’s a huge boon to have access to all those python models. Thanks for your work there!
Also have my eye on yota.jl
---
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [January 16, 2022, 9:31am UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/43 "2022-01-16T09:31:56Z")
> [@Akatz](#):
>
> If you’re restricted to a smaller set of centralized blessed ops, then you might as well use pytorch.
Having a list of operators with certain guarantees doesn’t restrict you in any way. Take mutation for example. Most AD frameworks either don’t favor or explicitly don’t support it (e.g. see the [long-standing issue in Zygote](https://github.com/FluxML/Zygote.jl/pull/75)). However, it doen’t mean you can’t use mutation in non-AD code (e.g. during data preprocessing) or hide it behind AD primitives (e.g. via pure `rrule`s).
This is in contrast to, say, JAX that has this [restriction](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#array-updates-x-at-idx-set-y) on the framework level. If you need a mutable array there, you have to fall back to NumPy (without GPU support) or some other library, ending up in a weird mix of technologies and endless conversions.
---
### Author:  [@Akatz](https://discourse.julialang.org/u/Akatz)
#### Post date: [January 16, 2022, 2:02pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/44 "2022-01-16T14:02:20Z")
I think having a list of blessed ops is a good stopgap for now, I agree.
And yea, data handling and non differentiable simulation code etc is much more pleasant in Julia
---
### Author:  [@JordiBolibar](https://discourse.julialang.org/u/JordiBolibar)
#### Post date: [January 17, 2022, 7:50am UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/45 "2022-01-17T07:50:38Z")
> For a couple of days I’ve been thinking of a **curated list of high-quality operators and language features** with particular guarantees
I fully support such idea! Moreover, I would add that the same should be done with **operators that are known to be buggy or problematic** (i.e. with open/known issues). From my experience using Zygote, this would have made my life so much easier. When you debug AD stuff in Julia, it is often very tricky to isolate which operator/line is crashing. With such a list, it would be so much easier to narrow down which are the potentially problematic operators/functions and directly go debug those. And of course, in order to make it less of a pain in the ass to maintain, only those tougher and long-standing bugs could be included. For new issues which have a potential easy fix in sight, they wouldn’t need to be added there for simplicity’s sake.
Such lists could be posted in the docs of each library and merged together in JuliaDiff to have a nice overview.
---
### Author:  [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [January 17, 2022, 9:19am UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/46 "2022-01-17T09:19:39Z")
May be rely more on [https://github.com/JuliaDiff/ChainRulesTestUtils.jl](https://github.com/JuliaDiff/ChainRulesTestUtils.jl) .
---
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [January 17, 2022, 12:58pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/47 "2022-01-17T12:58:14Z")
I’ve just created [**HQDL.jl**](https://github.com/dfdx/HQDL.jl) to start tracking the list of such operators. The package provides a macro `@inspect` (and a very similar `@analyze`, see their docstrings for the difference) that:
- checks if a function is even callable
- runs `ChainRulesTestUtils.test_rrule()` with `Array` and `CuArray` types, `Float64` and `Float32` precision
- checks if the function has docstring
The first [report](https://github.com/dfdx/HQDL.jl/blob/main/REPORT.md) obviously has many false positives, but it uncovers a few interesting observations. For example, many broadcasted activation functions from NNlib fail on `test_rrule()` for unclear reasons. Many other functions don’t define `rrule()` for broadcasting and thus rely on AD to handle it. Usually, AD is able to handle them as long as `rrule()` is defined for the element-wise function.
If this initial effort looks interesting for the community, I’ll add tests/benchmarks on several popular AD frameworks, interop with ONNX, as well as manual notes, e.g. mentions in the docs, links to known issues, etc. Also, so far I only added a handful of operators, so many more are to come.
---
### Author:  [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [January 17, 2022, 1:47pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/48 "2022-01-17T13:47:09Z")
This looks awesome.
---
### Author:  [@Akatz](https://discourse.julialang.org/u/Akatz)
#### Post date: [January 17, 2022, 4:10pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/49 "2022-01-17T16:10:29Z")
Amazing! What about adding a performance analysis JET.jl Pass? [https://github.com/aviatesk/JET.jl/blob/84aea0c97ecc83f955ab2fde455dce05d4d1736f/docs/src/optanalysis.md](https://github.com/aviatesk/JET.jl/blob/84aea0c97ecc83f955ab2fde455dce05d4d1736f/docs/src/optanalysis.md)
---
### Author:  [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)
#### Post date: [January 17, 2022, 6:11pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/50 "2022-01-17T18:11:47Z")
Great work and really good overview!
---
### Author:  [@longemen3000](https://discourse.julialang.org/u/longemen3000)
#### Post date: [January 17, 2022, 9:15pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/51 "2022-01-17T21:15:19Z")
> [@dfdx](#):
>
> If this initial effort looks interesting for the community, I’ll add tests/benchmarks on several popular AD frameworks, interop with ONNX, as well as manual notes, e.g. mentions in the docs, links to known issues, etc. Also, so far I only added a handful of operators, so many more are to come.
it seems similar to the tim holy efforts on invalidation, in the sense that you need a tool to diagnose what’s wrong before start fixing things
---
### Author:  [@rveltz](https://discourse.julialang.org/u/rveltz)
#### Post date: [January 18, 2022, 6:31am UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/52 "2022-01-18T06:31:22Z")
Dont we need a big library “a la `Plots.jl`” with different backends like
- flux
- knet
- yota
…
?
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [January 18, 2022, 6:52am UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/53 "2022-01-18T06:52:20Z")
> [@patrick-kidger](#):
>
> If I want to do the equivalent of PyTorch’s `detach` or JAX’s `stop_gradient` , how should I do that in Flux?
It might be helpful if there were a table showing each library’s name for common functions.
---
### Author:  [@Samuel\_Ainsworth](https://discourse.julialang.org/u/Samuel_Ainsworth)
#### Post date: [January 19, 2022, 1:25am UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/54 "2022-01-19T01:25:05Z")
> [@Samuel\_Ainsworth](#):
>
> At this point just about every last drop of performance has been squeezed out of pytorch/TF/jax in the “conventional” large models space.
Right after I say this 😛 [https://twitter.com/aleks\_madry/status/1483523047273512978](https://twitter.com/aleks_madry/status/1483523047273512978)… Would be interesting to try to replicate ffcv in Julia with MetaTheory.jl and native JIT compilation. That would really turn heads if it was actually faster!
---
### Author:  [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)
#### Post date: [January 19, 2022, 12:06pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/55 "2022-01-19T12:06:13Z")
Crazy speed improvement on their end. Gotta check this out 😌.
---
### Author:  [@laut](https://discourse.julialang.org/u/laut)
#### Post date: [February 2, 2022, 5:01pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/56 "2022-02-02T17:01:31Z")
> [@ChrisRackauckas](#):
>
> Flux is just the most complete in terms of kernels that exist, but its style irks me. I wish there was a Flux which did not use implicit parameters and instead used explicit parameters.
This quote makes me happy.
**kwargs \> pargs** all day every day.
---
### Author:  [@cscherrer](https://discourse.julialang.org/u/cscherrer)
#### Post date: [April 26, 2022, 3:10pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/57 "2022-04-26T15:10:55Z")
> [@ChrisRackauckas](#):
>
> Flux is just the most complete in terms of kernels that exist, but its style irks me. I wish there was a Flux which did not use implicit parameters and instead used explicit parameters.
In case you haven’t seen it,
> **[GitHub - LuxDL/Lux.jl: Explicitly Parameterized Neural Networks in Julia](https://github.com/LuxDL/Lux.jl)**
>
> Explicitly Parameterized Neural Networks in Julia. Contribute to LuxDL/Lux.jl development by creating an account on GitHub.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [April 26, 2022, 3:17pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/58 "2022-04-26T15:17:34Z")
Oh I know, I’m co-author on the paper it’s being written for 😅. It’ll get renamed and stuff first though, so it’s ready for a slack share but I wouldn’t throw it to twitter and everything yet.
---
### Author:  [@gvijqb](https://discourse.julialang.org/u/gvijqb)
#### Post date: [June 27, 2022, 7:13pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/59 "2022-06-27T19:13:00Z")
Just chiming in to see if the ONNX.jl progress is coming along. It’d be a huge boost to the entire Julia community I feel. Making the transition a lot easier to switch.
---
### Author:  [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [June 27, 2022, 8:33pm UTC](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385/60 "2022-06-27T20:33:29Z")
We’ve tagged a new release of ONNX.jl recently. Although it doesn’t have direct integration with Flux yet, it can already load and execute a number of popular operators.
At the moment, the main issue with further progress is the lack of workforce - around 80% of pending operators are trivial to implement. But maintainers typically have 4-6 other projects to care about, so `number_of_hands * fraction_of_time` is still pretty small.
(Perhaps I need to add a section for new contributors to the README 🤔 - Edit: [done](https://github.com/FluxML/ONNX.jl/blob/master/CONTRIBUTE.md))
[Previous page](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385.md?page=2)
[Next page](https://discourse.julialang.org/t/state-of-machine-learning-in-julia/74385.md?page=4)