# Lilith.jl is now called Avalon.jl

**URL:** <https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621>\
**Category:** Machine Learning\
**Created:** [December 10, 2020, 5:30pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621 "2020-12-10T17:30:18Z")\
**Posts on this page:** 12\
**Page:** 2

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [December 18, 2020, 11:30pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/21 "2020-12-18T23:30:12Z")

</div>

> [@Tomas\_Pevny](#):
>
> I would actually very much like to understand, why Iota is more performant that Flux. Where is the secret sauce?

I think the right question is why Flux is slow(er) for your use case. With all primitives like array operations and memory management on GPU already optimized by underlying frameworks, all libraries should perform approximately the same unless there’s a specific issue. It may be double conversion between `Float32` and `Float64`, or one slow adjoint, or unrecognized optimization. I think digging into your current implementation and profiling every piece will give you more benefits in less time than switching to another AD.

Yet, ChainRules is definitely on my list, so hopefully at some version switching between AD implementation will indeed be as easy as for BLAS 🙂

---

<div class="post-metadata">

**Author:** ![Marc.Cox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marc.cox/32/7514_2.png) [@Marc.Cox](https://discourse.julialang.org/u/Marc.Cox)\
**Post date:** [December 18, 2020, 11:42pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/22 "2020-12-18T23:42:05Z")

</div>

Avalon looks great ! I’m really looking forward to trying it out !

And I agree it’s important to consider **the “need for speed”** in both Production execution , and supporting the software development lifecycle. So I looked for and found the NewPkgEval.jl package to help Developers maintain backward compatibility and detect early any “build breakers” before they impact Multiple Package Integration Test/IT environments.

For your consideration it appears **NewPkgEval.jl** has methods that could automate parts of your **backward compatibility tests** ; NewPkgEval.jl @@ [https://github.com/JuliaCI/PkgEval.jl](https://github.com/JuliaCI/PkgEval.jl) automatically obtains multiple versions of Base Julia e.g. v1.05, v1.4.2, v1.5 etc. locally for you and provides “ … the following commands to run the tests of a list of packages on a selection of Julia versions : …”

IOW \>\> NewPkgEval.jl Helps \*\*Answer the burning Question : \*\*  
**Why does my package fail?**  
If you want to debug why your package fails, it’s probably easiest to use an interactive shell:  
julia\> using PkgEval

Ps\> And Yes, always ask about the free coffee and Ping Pong tables upfront ; because if they say no to ^that^ then “Free massages” is probably out of the question. 😉

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [December 19, 2020, 12:11am UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/23 "2020-12-19T00:11:26Z")

</div>

> [@dfdx](#):
>
> Large projects like PyTorch with huge user base and backed by multi-billion companies can focus on hundreds of things at the same time, but both - Flux and Avalon - are quite tiny and thus have to choose areas to spend most times on.

Of course, and there’s something to be said for being more proactive about putting warning signs up on bitrotted projects so that people understand what is actively maintained vs obsolete or just an experiment. I don’t want to tread on any feet here, but ONNX.jl, is kind of the poster child for that in FluxML right now.

> [@dfdx](#):
>
> Consider higher-order derivatives…That’s a huge time investment, and without clear benefits most likely not worth it.

That point was a bad distraction on my part, but the first one about missing dropout etc. still stands. Per your point though, that’s not a indictment of Avalon for being “unsuitable” for production, but rather an acknowledgement that it has a specific focus and that “production” (or applied research, in my case) requirements can be somewhat variable depending on the user.

I think the silver lining here is that not drowning in resources incentivizes the ecosystem to pool them instead of creating a separate stack for each framework. Hence why everyone is able to profit off NNlib, CUDA, ChainRules and the like. This even applies beyond the code level: did you know we had a lengthy [Github thread](https://github.com/FluxML/ML-Coordination-Tracker/issues/10) and multiple Zulip topics about ONNX import/export?

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [December 19, 2020, 10:56pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/24 "2020-12-19T22:56:09Z")

</div>

> [@Marc.Cox](#):
>
> For your consideration it appears **NewPkgEval.jl** has methods that could automate parts of your **backward compatibility tests** ; NewPkgEval.jl @@ [https://github.com/JuliaCI/PkgEval.jl](https://github.com/JuliaCI/PkgEval.jl)

This looks interesting, thanks! Usually I check the status of the package in all supported Julia versions in CI, but it doesn’t cover GPU stuff, so testing locally can be a good option.

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [December 19, 2020, 11:34pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/25 "2020-12-19T23:34:17Z")

</div>

> [@ToucheSir](#):
>
> That point was a bad distraction on my part, but the first one about missing dropout etc. still stands. Per your point though, that’s not a indictment of Avalon for being “unsuitable” for production, but rather an acknowledgement that it has a specific focus and that “production” (or applied research, in my case) requirements can be somewhat variable depending on the user.

Exactly. Frankly speaking, I can’t recommend neither Flux, nor Avalon as the main deep learning library for someone in the industry (just yet), and not because dropout or transposed convolutions are missing, but because there are still too many bugs and caveats. Sometimes these issues come from third party libraries and have quite long way to fixes (e.g. like [this bug](https://github.com/JuliaGPU/CUDA.jl/issues/592) in CUDA.jl), sometimes they hit corner cases and take weeks to fix (e.g. [this one](https://discourse.julialang.org/t/increment-elements-of-array-by-index/49694)). But it’s part of infrastructure maturing - when GPU stuff, web programming, API clients, big data tools, etc. are ready, we will already have ML kitchen in a good state to finally replace Python.  
What is good about existing deep learning libraries in Julia is that they are already suitable for certain tasks (e.g. I used Avalon extensively for my representation learning experiments, some of them can be found in model zoo) and if something is missing, it’s usually not too hard to add it (e.g. I’m currently working on Transformers which require at least `Embedding` layer, so that’s my next goal).

> [@ToucheSir](#):
>
> This even applies beyond the code level: did you know we had a lengthy [Github thread](https://github.com/FluxML/ML-Coordination-Tracker/issues/10) and multiple Zulip topics about ONNX import/export?

No I didn’t, thanks for letting me know! It will definitely influence my work in ONNX branch.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [December 20, 2020, 1:09am UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/26 "2020-12-20T01:09:52Z")

</div>

> [@dfdx](#):
>
> No I didn’t, thanks for letting me know! It will definitely influence my work in ONNX branch.

Awesome, I could’ve sworn I saw you on the GH thread but must have confused it with Discourse. Either way, things might be ramping up on the [Flux end](https://github.com/FluxML/Flux.jl/issues/1431). It would be great to get your input to see how much could be made framework agnostic!

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [December 20, 2020, 3:40pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/27 "2020-12-20T15:40:37Z")

</div>

> [@ToucheSir](#):
>
> Either way, things might be ramping up on the [Flux end](https://github.com/FluxML/Flux.jl/issues/1431).

Looks hot! I’m diving in.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [March 2, 2021, 3:37pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/28 "2021-03-02T15:37:05Z")

</div>

> [@dfdx](#):
>
> Avalon follows PyTorch API pretty closely, so usually translating a new model to it takes half an hour

Is that’s what you mean with “ **interoperability with existing DL frameworks** ”? You may want to spell it out in the docs. And since plural what other? ONNX? And/or PyTorch Lightning? I’m not up-to-speed on the latter, or if some Julia package corresponds to it. In general, see also:

> [@How do Julia packages, e.g. Flux and Knet correspond to Python ecosystem](https://discourse.julialang.org/t/how-do-julia-packages-e-g-flux-and-knet-correspond-to-python-ecosystem/55147):
>
> I’m not so much asking if they are better, just some mapping, and trying to get an overview of the mainstream and where there might be big holes in the Julia ecosystem. E.g. Pytorch Lightning corresponds to Keras, and to what in Julia? See e.g.: [Converting From Keras To PyTorch Lightning | by William Falcon | Towards Data Science](https://towardsdatascience.com/converting-from-keras-to-pytorch-lightning-be40326d7b7d) They are higher-level interfaces to Pytorch and TensorFlow, those are the most mainstream low-level packages right? There’s also MXNet, and Julia already officially p…

Since you link to [vision] transformer, would your package be best to replicate BERT-models, GPT-3, or Google’s even larger Switch Transformer model? Since it’s sparse, would that be a hindrance?

Google’s 1.6 trillion parameter model:

> **[Google trained a trillion-parameter AI language model](https://venturebeat.com/ai/google-trained-a-trillion-parameter-ai-language-model/)**
>
> Researchers at Google claim to have trained a natural language model containing over a trillion parameters.

> **[2101.03961.pdf](https://arxiv.org/pdf/2101.03961.pdf)**
>
> 1273.59 KB

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [March 2, 2021, 7:26pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/29 "2021-03-02T19:26:26Z")

</div>

> [@Palli](#):
>
> And since plural what other? ONNX? And/or PyTorch Lightning?

Not yet, but both of these are indeed on the roadmap.

> [@Palli](#):
>
> Since you link to [vision] transformer, would your package be best to replicate BERT-models, GPT-3, or Google’s even larger Switch Transformer model?

I think for such large-scale models with only known and well-optimized layers it doesn’t really matter what framework you use - they all will have the same size and approximately the same speed. One thing I can promise about Avalon (when it gets to its full vision) is that for any recent and more or less popular model you will be able to find existing PyTorch implementation, translate it to Avalon in under an hour and load pretrained weights (if any) via ONNX.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [March 3, 2021, 12:10am UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/30 "2021-03-03T00:10:57Z")

</div>

Right now it’s not possible to replicate such large models in _any_ Julia library (even Flux + Transformers.jl) because none of them support distributed training. It’s being worked on though!

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [March 3, 2021, 12:45pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/31 "2021-03-03T12:45:46Z")

</div>

Given the estimated [cost](https://www.reddit.com/r/MachineLearning/comments/h0jwoz/d_gpt3_the_4600000_language_model/) of these models, adding distributed training capabilities doesn’t seem to be the biggest problem 🙂

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [March 3, 2021, 7:08pm UTC](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621/32 "2021-03-03T19:08:47Z")

</div>

Right, but there’s definitely a continuum between say, AlexNet and GPT-3. Also worth considering that many folks are working with multi-GPU systems where each card has a limited amount of VRAM. Even something as “mainstream” as training a ResNet on ImageNet becomes significantly more painful when you’re forced to run it with a small batch size on a single device.

[Previous page](https://discourse.julialang.org/t/lilith-jl-is-now-called-avalon-jl/51621.md?page=1)