next week I am once again giving the Julia Zero2Hero workshop at the University of Exeter (now updated to Julia 1.9, including a Makie tutorial!).
As I am advertising Julia more and more, I constantly have the same question being asked at me “why Julia?”. Originally, I had information in the aforementioned workshop, but that information was becoming more and more extensive.
I’ve decided to split it off into a README repo, whose purpose is to have a well-organized, well-formulated answer to “Why Julia” (in particular in the academic context).
It would be great if you have a look and correct me in places that you think I say incorrect statements, or perhaps I am claiming too much. While this post reflects my opinions, I want it to be as factual as possible nevertheless.
Furthermore, feel free to contribute more detailed advantages of Julia in the last section of the document (the detailed bullet point list), assuming that you share my views on Julia
Very cool, I actually need something like this to explain why Julia is worth it at a meeting
This is something I think it could be improved:
For 5th section:
Julia is ahead of Pytorch
Pytorch is very advanced and on Discourse there are a lot of discussions where people advice to still use Pytorch for traditional workflows. So I wouldn’t say so. Maybe more that it is keeping up with it for traditional workflows and has advantages when you want to do something custom like writing a kernel yourself. See Is it a good time for a PyTorch developer to move to Julia? If so, Flux? Knet?
For 6th section:
Julia is written in Julia
This is more of a title issue than anything else since on the text you don’t mention that the compiler is written in itself, but the title hints at this from a user perspective. Consider that most of Julia is written in Julia, but there is a lot of C and other languages in the main repository. To be honest, also CPython is written mostly in CPython if you look at the composition of programming languages used in it. But surely Julia has standard libraries + Base written entirely in Julia, which is cool. See How is Julia written in Julia?
I would quibble with some of the claims in the repo. In particular
I think maintainability (debuggability, static analysis, stability, tooling, ease of refactoring) is another major core aspect of a programming language - arguably more important than ease of writing. You only write a program once, then you maintain it for years.
I would stress that the “two language problem” refers specifically to low vs high throughput languages. If your problem with Python is not its speed but its latency, then Julia will do not better, and probably worse. If your problem with Python is problems with maintenance, then Julia will do no better, either.
I would caviat the claim that Julia is written in Julia. Much of the compiler, the threading runtime, the GC, symbols and strings are written in C/C++.
I would not say that Julia’s package ecosystem is top-of-the-class in many scientific fields. Maybe a handful? I would bet that the ecosystems where Python outshines Julia outnumber the opposite by at least 10:1.
Thanks a lot for the feedback Jakob, that’s the kind of discussions I need!
I agree that maintainability is important, but I still stand that in academic usage, the other 4 reasons I list are more important. Speed of writing is by far the most important in my academic experience. The majority of scientists do no need to develop a software library to do their jobs. Thus, I would argue that “You only write a program once, then you maintain it for years.” is not the common case in academia. The common case is that indeed you write a script once to produce a figure for one paper and then you may likely not reuse it again.
That is correct, latency is an issue, but it appears unrelated to the two language problem. I don’t see any obvious way they relate. I believe that in the academic circles the “two language problem” is well established to refer to what you also state, hence I think it is not necessary to explain it further in the document.
100% right, user Tortar raised the same issue, and I corrected the repo accordingly!
“I would bet that the ecosystems where Python outshines Julia outnumber the opposite by at least 10:1.” I would need data to believe that actually. I would believe it blindly 2-3 years ago, today not so sure. But, sure, phrasing it more diplomatically is a good suggestion. I’ve changed it to “some”.
To me, one important thing is the quality of packages. Examples:
As late as 2010, the Optimization TB in MATLAB gave a wrong/very inaccurate solution to unconstrained QP problems. I got much more accurate solution by implementing the Lagrange multiplier method myself (which is trivial). Of course, this particular problem has been solved by now in the O TB, and I have no reason to doubt the quality of commercial TBs from MathWorks today, but quality is important.
When I tried to learn Python in 2011-12, one package I tested was an optimization package written in pure Python. The quality was less than impressive… to find the optimal solution to the (2D) Rosenbrock banana function (a standard test case), I had to give an initial guess that was within 10% from the optimal solution. Of course, linked in Fortran/C code in SciPy were much better. And I’m sure a lot of Python code is of qood quality. Still, quality is important.
My impression is that a large number of Julia packages are of very good numeric quality. To me, that is important. I also like (i) the syntax of the language, and (ii) that many developers are quite young, leading to a dynamic development environment with a certain level of future.
Multiple and dynamic dispatch complement each other, but they are quite different things. Dynamic dispatch leads to choosing the right method at runtime, so we can implement predicate dispatch or combine code with abstract (non-inferred) and concrete (inferred) types. It is based on type inference, method specialization, method call inlining:
Metaprogramming: Generated functions - we can use type information at parse time and generate specialized code Abstraction in technical computing pages 78-79
We can leverage compiler complexity and solve problems just adding new syntax or macros that solves our problem before compiler stage, e.g. Loop fusion: More Dots: Syntactic Loop Fusion in Julia
4. Composability
I would mention a bunch of interesting solutions in platform and package manager like:
Persistent binary artifact storage + Yggdrasil + build system for any platform combinations: Home · BinaryBuilder.jl
Clang.jl - automatic C wrapper code generation (used in BinaryBuilder) that helps for many third-party libraries to just work
My view of the two language problem is much more pedestrian. I never prototyped things in python to later translate to Fortran. When writing numerical code Fortran was good enough from start for me.
But then I had to distribute my code, even to my students, which would want to produce plots, data frames, statistics, etc. For that the solution was a patchwork of bash, python, TCL, web interfaces, etc. The best possible scenario was when I wrote a python interface to the code. And from there to actually have a standard distributed package there was still a barrier which I never overcame.
With Julia not only I write the numerical code, but make it into a package, write the docs, thus making many original throw away scripts useful tools for my group and, hopefully, to other people. And the students are able to use the packages and do all further analysis (non critical in terms of performance) within Julia. I am even using Julia as an interface to distribute cross platform binaries of packages.
I do have now a commitment with maintaining packages for long, but that’s a good thing, for my research group and for the impact of my work.
I’d add that while writing Julia code with the highest possible performance is not super easy or clean anymore, people can package these optimized pieces of code up into much smaller libraries which others can then plug into their own code to benefit from that. For example in numpy, you opt into the whole ecosystem or not, you don’t really pick an algorithm here or there because it all assumes working with the framework. So it’s nice that their algorithms are also tuned to highest performance, but they are not so highly composable. In Julia, that’s different. Usually, only the scalar kernel functions have to be tuned, the rest can be orchestrated with broadcasting or simple map calls in most cases. I think that lowers the barrier for contributing.
I agree a lot with this in general, but want to add a caveat. In Python, interop is better for the biggest APIs and worse for the rest. (I think a lot of this is because Python’s interfaces are a very recent feature, with the limited adoption that entails.) ~99% of all code will work if you pass a Pandas DataFrame through it, or anything else that implements the Pandas API. On the other hand, almost nothing works with Polars (which explicitly chose not to follow the Pandas API because it leads to slow, difficult-to-read code).
With Julia, OTOH, if you try passing something through a function (like a named array), there’s an 80% chance it will work. This is true regardless of the implementation you pick, because Julia has built-in (unenforced) “interfaces” that are widely used (e.g. the array interface); but because they’re undocumented and unenforced, there’s always a very good chance your package just won’t work with it.
It’s not as fundamental as the other points (and it might just be personal preference), but in my continuing Julia learning journey, I found that transitioning from Jupyter to Pluto helped greatly; it’s where I go to work things out.
First of all, great content. I’m glad there’s an up-to-date and clear description of the arguments to use Julia written for its largest audience. That said, to me it does come off quite zealous, and I believe that you would convince more people to use Julia if the language used in the doc was tempered a little.
For example, you begin the disclaimer with the word “clearly”, which sets the tone for the rest of the doc. The same disclaimer then says that “these are opinions, but they’re based on facts”, but at one point your “proof of claims” is one of Julia’s creators speaking at a Julia conference about why Julia is so great. I don’t think many who aren’t already agreeing with you will be convinced by this.
Additionally, I think when talking about advanced topics, especially for a relatively lay audience, it’s important to give, or link to, definitions of concepts that you talk about. There are a few places where scientists might get lost or not understand why something is an advantage because they’re not familiar with the concept e.g. what do you mean by “innovative progress such as automatically understanding when statements end”.
Idk, I just feel a little sidelined when reading this from a Python perspective, as if the Julia community thinks Python sucks and I’m dumb for using it when there are many reasons why one might prefer using Python. This comes partly from the tone e.g. your claim that the document is your opinion is not necessarily substantiated by your use of the phrase “thankfully, Julia is objectively the best tool for…” :')
I hate to be a bummer. I think it’s truly amazing that this document is available and I respect the amount of sustained effort that goes into writing such a document. That said, take what I say with a grain of salt: I have a track record for this kind of post
Thanks about your concerns and criticism! Yes, about some parts you have convinced me! I have reduced the “zealousness” in a couple of key points. One needs to be careful to not try to enforce things as obvious when they aren’t:
But overall I want to keep my style and personality intact. And zealousness and passion are separated by a very thin line after all, and I’m a passionate person about some things…!
More importantly than keeping my own tone, I am concerned by a couple of your comments, for example:
And what “tone” does it set? Is it a “bad” tone? Your phrasing makes it sound like it sets at minimum a “non-positive” tone. Yet, I must have read by now 100+ textbooks that start sentences all the time with the word “clearly”. Probably this is how I learned to speak this way as well. Should I be using this as ammunition to claim that their “tone” is “non-positive” and should be changed?
Well, not really. I just accept that in today’s world it is impossible to make any meaningful statement without offending someone, and just carry on knowing that someone will be offended by what I write…
I strongly object!!! You are referring to Stefan’s talk “The unreasonable effectiveness of multiple dispatch”. This talk is not at all why Julia is great. It is why multiple dispatch is great. Not only that, but the talk is genuinely factual. The talk says that MD is great because
It allows adding new methods to existing types
It allows adding new types to which existing methods apply
It allows for the incredible composability Julia packages have that we have not seen in any other language
Why have we not seen that in other languages? Well, the talk proceeds to show, objectively and with real code, how this system can’t be done in neither Python or C. So yeah, I would say this is a very objective case of the greatness of the system. Julia is great because it uses this great system. Not “Julia is great”, because “a creator said so”. To me it seems that your comment dismisses all this objective and informative debate just because it is stated by one of the creators of the language.
I mean that you don’t have to end lines with ; like in C or use \ to continue in the next line like in Python. This is explained later in “Details” section:
Additionally, the modern Julia syntax parser eliminates the use of many “decorators” such as ending lines with ; or requiring indentation to denote code blocks, as it understands automatically when commands start and end.
One has to strike a balance between swarming with details and making high level statements. It is not always easy to find this balance. Especially given that the point of balance is different for different people. If there are other statements that you think are unclear let me know!
I have to admit, I think this is “strawman-ing” me. Even worse, it is referring to the Julia community as a whole, which is odd, because surely at least the fact that this document is not written by an entire community should be clear. Although, I do agree with you that I should be making clear what things are my opinions so I did rephrase the sentence you cited. In it’s current form of the document, I can’t find anywhere in statements that allude to someone being stupid, so please substantiate the claim if it is still there.
Thanks for the reply and for taking my comments on board. That said I think you should have kept the wording around Pkg.jl, which is objectively the best package manager :S
To add some context to my post, while I enjoyed reading the manifesto, I couldn’t help but think about how someone new to the language might get the impression of a perfect language that immediately makes them a coding wizard. Now, this is an unrealistic expectation for anyone to hold, but when someone doesn’t know much about programming hears such a glowing review it can really set their expectations into overdrive and can lead to disappointment.
Don’t read too much into it, I think it’s more my instinctive reaction when I read something written in this tone because of what I’ve read in the past. It doesn’t actually imply anything about the text. That said, for this specific example, when you say “clearly, this is my opinion” I get the impression that some opinions may be asserted as facts with the disclaimer used as a shield against criticism (which you’re not!). This is because of the word “clearly”, which (to me) makes the degree to which you’re anticipating criticism about this topic is suspiciously strong
Sure, but someone who is offended shuts down their logical brain and starts looking for other things to disagree with you on. I think you’re less likely to convince people in this way.
It’s just a question of optics imo. I agree that the talk is great and states objective advantages of multiple dispatch, but how can some scientist know that? I think someone unfamiliar with the topic sees what I said, and concludes that you’re basing your opinion on Stefan’s opinion.
I apologise if I’m being overly negative on this. It’s kinda hard to find an explanation of multiple dispatch from people who aren’t Julia adjacent, but it might help to find a source where the intended audience is more clearly the general programming public. It kinda looks like Stefan is “preaching to the choir” from an outsider’s perspective.
I forgot you had to do that! Eww.
I apologise, that wasn’t my intention at all. On the community aspect, I guess what I’m implying is that we are representatives of the communities that we are members of. The manifesto may not be “endorsed” by the community (being linked somewhere etc.), but it is starred by other members of the community, which is in its own way a form of endorsement.
On the “feeling stupid” part, I kinda ran out of steam, got tired, and just wanted to get my thoughts down. Not a great thing to do when writing a permanently visible forum post tbh. Ultimately, how words make people feel is a personal thing and I just happened to feel a little uneasy by some of the language in the manifesto. This is largely cultural, but I’ve also had to work on changing my own language throughout my life to be more accommodating and so I feel like I can recognise things that might be misconstrued or read in the wrong way.
Julia compiles to machine code that is routinely as fast as C/FORTRAN
“Julia is a compiled language, and optimized Julia code is typically about as fast as the corresponding C/FORTRAN” would be a more realistic description IMO. Julia code routinely written by an average scientist is probably somewhere between Python and C on the execution speed (which is though sufficient in many cases)
This phrase is difficult to understand, and the fact that you (try to) explain it later doesn’t help much. Furthermore I do not think that having or having not indentations and end statements etc. can count as “innovative progress” nowadays.
I am with @jacobusmmsmit also on other points mentioned. Sure, any Manifesto is per definition opinionated, but I do think too much overselling is dentrimental to your/our goals.
I don’t agree. Some scientist that’s used to write Fortran or C will have an easier life writing code with the same performance in Julia, as routine. Particularly for numerical code, just not using global variables and taking care of allocations in hot loops gets one there. The parts that are similar to Python are just those where performance doesn’t matter, and many of those are a nightmare to write in C, Fortran, etc.
“Optimized codes”, if that means the insane exercises of benchmark games, have little to do with routine programming and are hard and exceptional in all languages.
