# Why are compiler devs in such high demand for Julia?

**URL:** <https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420>\
**Category:** General Usage\
**Tags:** question, compilation\
**Created:** [May 26, 2023, 2:25am UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420 "2023-05-26T02:25:21Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![anon56330260](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@anon56330260](https://discourse.julialang.org/u/anon56330260)\
**Post date:** [May 27, 2023, 3:59am UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/21 "2023-05-27T03:59:22Z")

</div>

> [@ToucheSir](#):
>
> “we’re waiting for this compiler thing to land”

This also happens in other languages, not just Julia. For example, C++ adds move semantics (C11) and concepts (C20), which are major updates to the language. Rust’s ownership system is improved from time to time, to handle cyclic and closure better. Java also introduces patterm matching in newer version jdk.

Maybe you mostly interactive with some experienced Julia users so the observation is biased.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [May 27, 2023, 4:08am UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/22 "2023-05-27T04:08:41Z")

</div>

How many libraries stopped feature development until those changes landed? How many new libraries were created well before they landed in anticipation of them landing? Even for something as disruptive as async in Rust, I don’t believe there was any appreciable dip in activity in web or IO library development because people were waiting for it to be implemented. If anything it was the exact opposite, which is part of why the library ecosystem transition from a pre-async to post-async world in Rust took so long.

---

<div class="post-metadata">

**Author:** ![Tarny\_GG\_Channie](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Tarny\_GG\_Channie](https://discourse.julialang.org/u/Tarny_GG_Channie)\
**Post date:** [May 27, 2023, 4:42am UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/23 "2023-05-27T04:42:32Z")

</div>

This might be true to a certain scale, but I still believe in dynamic languages because of generics. Dynamic languages are more economical in writing generics, which, when the codebase size approaches infinity, the most important factor is generic code. X% of code written and used as a generic library means X% of code as a library that gets used and maintained (as the codebase size approaches infinity) infinitely many times. On a more practical note, applications are smaller as more functionalities are implemented by composable libraries, becoming like a network.

Yes, you’re talking about a LARGE codebase so I am allowed to assume the codebase size approaches infinity.

---

<div class="post-metadata">

**Author:** ![Tarny\_GG\_Channie](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Tarny\_GG\_Channie](https://discourse.julialang.org/u/Tarny_GG_Channie)\
**Post date:** [May 27, 2023, 5:22am UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/24 "2023-05-27T05:22:15Z")

</div>

In addition, we need to understand why static typing works well. First, imagine you call a functionality, you could get

1. It was supported.
2. It was not supported and not caught
3. It was not supported and caught

Static typing improves 3, but Julia comes up with a new possibility.  
It just works, improving 1.  
That’s what Julia’s strength is.

---

<div class="post-metadata">

**Author:** ![Tarny\_GG\_Channie](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Tarny\_GG\_Channie](https://discourse.julialang.org/u/Tarny_GG_Channie)\
**Post date:** [May 27, 2023, 7:25am UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/25 "2023-05-27T07:25:43Z")

</div>

Back onto the topic, there are two camps here, one believing there is more feature in Julia using the compiler, and one suggesting that compiler development is just more visible for Julia.

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [May 27, 2023, 1:23pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/26 "2023-05-27T13:23:26Z")

</div>

I find that a blocker for me in getting started contributing to core things is finding the entry points for the code. If I want to look at “how compilation works”, I can find very complicated flow charts that explain the algorithm which are helpful in understanding it theoretically. However, I find it hard to find the literal functions in Base that get called when I execute a function for the first time. I spend a bunch of time reading source code, piecing things together, and then run out of time/energy to actually think about contributing.

I wonder if others would find it valuable to have a JuliaCon presentation that walks through “How [Complication or GC or whatever] is implemented” that doesn’t attend so much to the exact logic of the algorithm, but instead tours the code and shows where one should look for different aspects of the processes.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [May 27, 2023, 1:34pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/27 "2023-05-27T13:34:48Z")

</div>

I often find Cthulhu pretty good at this since if you know the entrypoint, it lets you go through and find the functions that get called (and jump into an editor from there).

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [May 27, 2023, 1:44pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/28 "2023-05-27T13:44:03Z")

</div>

That’s a great idea! Maybe the presentation is actually just a lightning talk called “So you want to contribute to Base?” And it’s a list of entry point functions for various aspects of base, a brief demonstration of a workflow using Chulhu, and a primer on the etiquette of opening an issue/PR.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [May 27, 2023, 1:52pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/29 "2023-05-27T13:52:26Z")

</div>

When you speak of packages like “Zygote, Diffractor, and Enzyme” then demand for people isn’t strictly about Julia, rather (a very specific; and tiny(?) part of) Julia ecosystem? And I believe we get away with way fewer people than similar efforts for similar tools usable from Python.

If you’re talking about:

> **[GitHub - JuliaSIMD/LoopModels: "Full speed or nothing." - James Hetfield](https://github.com/JuliaSIMD/LoopModels)**
>
> "Full speed or nothing." - James Hetfield. Contribute to JuliaSIMD/LoopModels development by creating an account on GitHub.

Then it’s a C++ project actually…, but done by a great Julia developer, for Julia. And what he did before was to make Julia world-class speed, in practice, what C++ and Fortran couldn’t. I.e. rivalling hand-tuned assembly OpenBLAS.

Languages that do not include a compiler at runtime, like C, C++, Go, Java and Fortran, do not have to improve precompilation, because there it’s just compilation. And decades of work have gone into that (for C/C++ and LLVM). We reuse most of it, but have some specific problems. No language compares to Julia, Lisp and some others in a way, but none are as fast. Mojo is now rivaling Julia, but not really.

> Julia wants to define its memory model in order to support atomic operations

Isn’t it already defined? As same as for C++? You need this for concurrency, and not I think for e.g. Python.

I see most work not needing compiler devs, it’s only in very specific projects.

“ **Note that LLVM itself does not provide a garbage collector**. You must provide your own.”

I.e. Julia “must”, and did provide one, and it’s been improving, e.g. in 1.9, and a parallel effort MMTK, that might get merged. I don’t actually think many people are working on the GC, just a handful I recall (I don’t have overview on MMTK). Other languages like Go, also made their own GC. Julia could have skipped having a GC, so “must” is strictly not true, and could maybe have reused a GC from other projects from the start.

---

<div class="post-metadata">

**Author:** ![wsmoses](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsmoses/32/26497_2.png) [@wsmoses](https://discourse.julialang.org/u/wsmoses)\
**Post date:** [May 27, 2023, 10:51pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/30 "2023-05-27T22:51:03Z")

</div>

[teasingly] that sounds like a great topic for a PhD thesis 🙂 [https://c.wsmoses.com/theses/Supercharging%20Programming%20through%20Compiler%20Technology.pdf](https://c.wsmoses.com/theses/Supercharging%20Programming%20through%20Compiler%20Technology.pdf)

---

<div class="post-metadata">

**Author:** ![martin.d.maas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/martin.d.maas/32/50964_2.png) [@martin.d.maas](https://discourse.julialang.org/u/martin.d.maas)\
**Post date:** [May 28, 2023, 10:25am UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/31 "2023-05-28T10:25:14Z")

</div>

> [@jpsamaroo](#):
>
> we need some way to encourage developers to gain the relevant experience to become compiler developers that know how to work on Julia and its libraries. Doing so involves a mix of things:
> 
> - Better developer documentation for Julia’s internals and library internals
> - Better inline code documentation in Julia’s compiler
> - Examples and tutorials for leveraging Julia’s codegen and reflection capabilities to do useful things like AD, “from scratch”
> - Examples and tutorials for doing development at the level of Julia IR and LLVM IR, from Julia
> - High-level and mid-level documentation of how Julia’s key features (JIT, multiple dispatch, etc.) work in practice

> [@mrufsvold](#):
>
> I find that a blocker for me in getting started contributing to core things is finding the entry points for the code. If I want to look at “how compilation works”, I can find very complicated flow charts that explain the algorithm which are helpful in understanding it theoretically. However, I find it hard to find the literal functions in Base that get called when I execute a function for the first time. I spend a bunch of time reading source code, piecing things together, and then run out of time/energy to actually think about contributing.

I agree 100%… I was just wishing that we had a “compiler tutorial”.

For what I understand (this topic is currently above my skill level) we can operate at different stages of the compiler pipeline:

- AST level with metaprogramming
- Before and after inference with tolls like [CodeInfoTools](https://github.com/JuliaCompilerPlugins/CodeInfoTools.jl) and [Mixtape](https://github.com/JuliaCompilerPlugins/Mixtape.jl).
- IR/SSA via ??
- LLVM via [LLVM.jl](https://github.com/maleadt/LLVM.jl)
- More things to know, perhaps?

I also understand there is also not one, but two major compiler pipelines: the native compiler, and GPUCompiler… however, I don’t really know why it’s not the same pipeline. There is also StaticCompiler built on top of GPUCompiler.

In my case, I’m not so interested in AD but rather in performance improvements related to memory management or being able to add features to StaticCompiler.

---

<div class="post-metadata">

**Author:** ![Tarny\_GG\_Channie](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Tarny\_GG\_Channie](https://discourse.julialang.org/u/Tarny_GG_Channie)\
**Post date:** [October 10, 2023, 12:37pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/32 "2023-10-10T12:37:07Z")

</div>

Oh, I just realized… Julia is still on LLVM version 15? It’s version 17 already. Where are we planning to upgrade? Should we wait for LLVM 18,19, or 20 and reserve compiler devs for some other tasks?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [October 10, 2023, 12:41pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/33 "2023-10-10T12:41:01Z")

</div>

Upgrade to 16 is in progress. LLVM major versions are breaking changes so it takes a non-negligable amount of work to update each time.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [October 10, 2023, 2:49pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/34 "2023-10-10T14:49:37Z")

</div>

> [@Tarny\_GG\_Channie](#):
>
> In other languages, I see the compiler as a static thing and libraries getting developed. Here, I see things generally used needing compiler devs to work. How did it turn out this way?

To a large part, I see this as the sign of a very good thing: The compiler/runtime is much more hackable. If you need something that is not supported, then it is feasible to add support.

Compare this to the JVM world (my current dayjob).

If I need something important in the language and propose to just add it to the JVM, then I get rightfully laughed out of the room.

Heck, it took years to get us finally upgraded from java8 (VarHandle FTW!). Who knows when we’ll upgrade to java21 (green threads FTW! Like julia had since 1.0…) and when Valhala lands (I’d guess 2030? struct support, like C or julia had from their very conception).

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 10, 2023, 3:31pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/35 "2023-10-10T15:31:21Z")

</div>

> [@Oscar\_Smith](#):
>
> Upgrade to 16 is in progress. LLVM major versions are breaking changes so it takes a non-negligable amount of work to update each time.

tangent, do you know why LLVM has to make breaking changes every few months?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 10, 2023, 4:02pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/36 "2023-10-10T16:02:29Z")

</div>

Because they can. They happen to have users who are willing and able to deal with constant breaking changes.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [October 10, 2023, 4:03pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/37 "2023-10-10T16:03:30Z")

</div>

> <https://github.com/JuliaLang/julia/pull/51591#issuecomment-1749512994>
>
> \> One general question, why not just skip LLVM 16 and go straight to LLVM 17?
> 
> …
> Are you volunteering to do that work? LLVM updates are tedious, fraught with risk and while it might be nice to skip an entire version you end up with more work and less ability to find out when something broke, because you can't bisect the middle.
> 
> If anyone wants to get involve getting the LLVM 16 build into Yggdrasil and looking at LLVM 17 in Yggdrasil would be a major contribution.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [October 10, 2023, 4:12pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/38 "2023-10-10T16:12:25Z")

</div>

One semi-serious reason is that it does a pretty good job of encouraging hardware vendors to upstream patches rather than fork LLVM.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [October 10, 2023, 5:40pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/39 "2023-10-10T17:40:54Z")

</div>

Fujitsu compiler is based on llvm 7 or so

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 10, 2023, 6:14pm UTC](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420/40 "2023-10-10T18:14:46Z")

</div>

that just makes the questions morph into “why so many patches need upstreaming are breaking changes”

[Previous page](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420.md?page=1)

[Next page](https://discourse.julialang.org/t/why-are-compiler-devs-in-such-high-demand-for-julia/99420.md?page=3)