This also happens in other languages, not just Julia. For example, C++ adds move semantics (C11) and concepts (C20), which are major updates to the language. Rust’s ownership system is improved from time to time, to handle cyclic and closure better. Java also introduces patterm matching in newer version jdk.
Maybe you mostly interactive with some experienced Julia users so the observation is biased.
How many libraries stopped feature development until those changes landed? How many new libraries were created well before they landed in anticipation of them landing? Even for something as disruptive as async in Rust, I don’t believe there was any appreciable dip in activity in web or IO library development because people were waiting for it to be implemented. If anything it was the exact opposite, which is part of why the library ecosystem transition from a pre-async to post-async world in Rust took so long.
This might be true to a certain scale, but I still believe in dynamic languages because of generics. Dynamic languages are more economical in writing generics, which, when the codebase size approaches infinity, the most important factor is generic code. X% of code written and used as a generic library means X% of code as a library that gets used and maintained (as the codebase size approaches infinity) infinitely many times. On a more practical note, applications are smaller as more functionalities are implemented by composable libraries, becoming like a network.
Yes, you’re talking about a LARGE codebase so I am allowed to assume the codebase size approaches infinity.
In addition, we need to understand why static typing works well. First, imagine you call a functionality, you could get
Static typing improves 3, but Julia comes up with a new possibility.
It just works, improving 1.
That’s what Julia’s strength is.
Back onto the topic, there are two camps here, one believing there is more feature in Julia using the compiler, and one suggesting that compiler development is just more visible for Julia.
I find that a blocker for me in getting started contributing to core things is finding the entry points for the code. If I want to look at “how compilation works”, I can find very complicated flow charts that explain the algorithm which are helpful in understanding it theoretically. However, I find it hard to find the literal functions in Base that get called when I execute a function for the first time. I spend a bunch of time reading source code, piecing things together, and then run out of time/energy to actually think about contributing.
I wonder if others would find it valuable to have a JuliaCon presentation that walks through “How [Complication or GC or whatever] is implemented” that doesn’t attend so much to the exact logic of the algorithm, but instead tours the code and shows where one should look for different aspects of the processes.
I often find Cthulhu pretty good at this since if you know the entrypoint, it lets you go through and find the functions that get called (and jump into an editor from there).
That’s a great idea! Maybe the presentation is actually just a lightning talk called “So you want to contribute to Base?” And it’s a list of entry point functions for various aspects of base, a brief demonstration of a workflow using Chulhu, and a primer on the etiquette of opening an issue/PR.
When you speak of packages like “Zygote, Diffractor, and Enzyme” then demand for people isn’t strictly about Julia, rather (a very specific; and tiny(?) part of) Julia ecosystem? And I believe we get away with way fewer people than similar efforts for similar tools usable from Python.
If you’re talking about:
Then it’s a C++ project actually…, but done by a great Julia developer, for Julia. And what he did before was to make Julia world-class speed, in practice, what C++ and Fortran couldn’t. I.e. rivalling hand-tuned assembly OpenBLAS.
Languages that do not include a compiler at runtime, like C, C++, Go, Java and Fortran, do not have to improve precompilation, because there it’s just compilation. And decades of work have gone into that (for C/C++ and LLVM). We reuse most of it, but have some specific problems. No language compares to Julia, Lisp and some others in a way, but none are as fast. Mojo is now rivaling Julia, but not really.
Julia wants to define its memory model in order to support atomic operations
Isn’t it already defined? As same as for C++? You need this for concurrency, and not I think for e.g. Python.
I see most work not needing compiler devs, it’s only in very specific projects.
“Note that LLVM itself does not provide a garbage collector. You must provide your own.”
I.e. Julia “must”, and did provide one, and it’s been improving, e.g. in 1.9, and a parallel effort MMTK, that might get merged. I don’t actually think many people are working on the GC, just a handful I recall (I don’t have overview on MMTK). Other languages like Go, also made their own GC. Julia could have skipped having a GC, so “must” is strictly not true, and could maybe have reused a GC from other projects from the start.
[teasingly] that sounds like a great topic for a PhD thesis 
https://c.wsmoses.com/theses/Supercharging%20Programming%20through%20Compiler%20Technology.pdf
I agree 100%… I was just wishing that we had a “compiler tutorial”.
For what I understand (this topic is currently above my skill level) we can operate at different stages of the compiler pipeline:
I also understand there is also not one, but two major compiler pipelines: the native compiler, and GPUCompiler… however, I don’t really know why it’s not the same pipeline. There is also StaticCompiler built on top of GPUCompiler.
In my case, I’m not so interested in AD but rather in performance improvements related to memory management or being able to add features to StaticCompiler.
Oh, I just realized… Julia is still on LLVM version 15? It’s version 17 already. Where are we planning to upgrade? Should we wait for LLVM 18,19, or 20 and reserve compiler devs for some other tasks?
Upgrade to 16 is in progress. LLVM major versions are breaking changes so it takes a non-negligable amount of work to update each time.
To a large part, I see this as the sign of a very good thing: The compiler/runtime is much more hackable. If you need something that is not supported, then it is feasible to add support.
Compare this to the JVM world (my current dayjob).
If I need something important in the language and propose to just add it to the JVM, then I get rightfully laughed out of the room.
Heck, it took years to get us finally upgraded from java8 (VarHandle FTW!). Who knows when we’ll upgrade to java21 (green threads FTW! Like julia had since 1.0…) and when Valhala lands (I’d guess 2030? struct support, like C or julia had from their very conception).
tangent, do you know why LLVM has to make breaking changes every few months?
Because they can. They happen to have users who are willing and able to deal with constant breaking changes.
One semi-serious reason is that it does a pretty good job of encouraging hardware vendors to upstream patches rather than fork LLVM.
Fujitsu compiler is based on llvm 7 or so
that just makes the questions morph into “why so many patches need upstreaming are breaking changes”