A monthly newsletter, mostly on julia internals, digestible for casual observers. A biased, incomplete, editorialized list of what I found interesting this month, with contributions from the community.
“Internals” Fora and Core Repos (Slack/Zulip/Discourse/Github):
- Julia 1.10 was released around Christmass. See the release highlights.
- It seems we all agree that it would be incredibly valuable to have better ways to call julia from inside python. This discourse thread has plenty of discussion on the topic, but specifically check out Chris’s comments which has a bunch of constructive suggestions towards the end.
- Rounding operations on Complex numbers (into presumably complex integers) are pretty difficult to do consistently. On the issue tracker folks have been discussing various ways to do it, and the dangers of being too permissive in what interfaces might emerge by accident. It is not feasible to summarize the entire story, as the thread is a bit meandering and has some contextual miscommunications, but a valuable read nonetheless. Arbitrarily, consider starting with this comment.
- A lot of ongoing discussion in various places on how to deal with asynchronous exceptions (interrupts from the outside or stack overflows, that can happen at much weirder ill-defined times compared usual exceptions). This is a good starting point for the discussion. There are comments with interesting references to how other languages deal with this as well.
- Now that we have a
publickeyword, people are thinking about how to better protect (or mark as sensitive) package internals, to avoid accidental dependencies on unstable implementation details. - A wonderful explanation of how “world splitting” used to be a great optimization in the compiler that can now lead to invalidating large amounts of code, cause recompilation, and de-optimization. The thread as a whole has other useful pieces of wisdom, but it is a bit meandering.
- A wonderful case study in fixing method invalidations and compilation latency (using Pkg.jl as an example). See this related older post on similar topics.
- An interesting discussion on the idiosyncrasies of computing type intersections (the set of types that are at the same time subtypes of two given types) in Julia. The operation is not commutative.
- Fun examples of how the update to a new LLVM is now changing the loop unrolling behavior (occasionally to detrimental effects) on slack and github.
Notes from other languages:
- With all the work on effect analysis in the julia compiler, it might be fun to see some similar work discussed in Rust.
- Julia is unsurpassed in its support for writing kernels for weird non-CPU hardware, but it is worth seeing what other communities are contributing to this topic as well: GPU programming in Chapel.
Core Julia Repos:
- If you are a developer running CI for your packages, consider turning on the new julia caching recipe for Github Actions. In my experience, for a repo where tests take around 10min,
julia-actions/cachemakes them 3min faster (compared to when one uses the github providedcacheaction). - If you are frequently moving between julia versions for the same project (e.g. as a developer), now you can have separate Manifest files for each julia version
- We just recently finally transitioned Julia to LLVM v16. A brave new contributor is attempting the LLVM v17 transition now. Check the llvm channel on slack for a cute story of their nerdsniping.
- Your error messages can be better if you use “error hints” – additional messages generated depending on properties of the thrown error. Check this recent example which provides a helpful message to distinguish
maxandmaximum. - The official download page for the language now specifically calls out juliaup as the most convenient installation tool.
- Steven has been spearheading the effort to ensure that every exported symbol has a docstring (a perfect place for a novice to contribute a patch to julia itself). Relatedly, now you can add such checks to your tests and CI thanks to new Base functionality.
- If you are doing a lot of metaprogramming you might like
remove_linenums!recently made public. - Wouldn’t it be convenient if we could type
using SomePackageWithLongName as ShortName. Lilith is working on it. - A neat example of a microoptimization in Base julia, drastically improving the performance of creating an empty Dict.
Dustbin of History:
- There is a story about how Fortran is better than C for scientific programming because Fortran arrays are guaranteed to not alias (to not overlap in memory), while working with a C array carried simply by a pointer does not let you guarantee that you are not also modifying some other array nearby in memory, so the compiler can not prove that values have not been modified. Julia has a long history of discussing aliassing including new ways to promise to the compiler that things do not alias (but please do not lie to the compiler). See 8087, 19658, and 31018, as well as UnsafeAssume.jl
Ecosystem Fora, Maintenance, and Colab Promises (Slack/Zulip/Discourse/Github):
- The autodiff zoo of tools in Julia has a newly updated and wonderfully detailed website showcasing the richness of autodifferentiation approaches available in the language.
- LoopVectorization.jl is being deprecated on julia 1.11 due to the accumulated maintenance burden. In no particular order see [github0], [github1], [github2], [slack1], [slack2]. One day LoopModels might become a better alternative.
- A reminder that DataStructures.jl really neads some tender love and care – it is fairly foundational for the ecosystem, very stable, but it needs a few minor hurdles dealt with before a long-term v1 is released. Please consider volunteering.
- Julia’s method specialization capabilities (compiling methods for each type it encounters) is fundamental to its amazing performance. However, there is code that is inherently type unstable for good reasons, which leads to incredibly bad performance (because of constant compilation of yet more specialized methods). With
@nospecializeinferyou can mark such situations, ensuring the compiler does not go wild. JLD2.jl is an example where we get multiple orders of magnitude improvement in performance (it is a serialization library). - A pair of posts on creating a 2D platformer game in Julia. Pretty wonderful introduction to the language. Do not forget to check out the video recording.
- We have mentioned Tidier.jl in the past – they are a wonderful datascience tool with similar design style and breadth of capabilities to R’s tidyverse. They made a recent “official” announcement to describe the entirety of their ecosystem.
- Oxygen.jl, a tool for creating web APIs, is seeing a new release, including a neat performance and utilization dashboard.
- LengthFreeStaticMatrices.jl is a very neat new take on Static arrays with potentially more pleasant type-specification API.
- A brief slack discussion on the status of AbstractDifferentiation.jl (a unified autodiff interface to the many Julia autodiff backends) and how plausible it is that Enzyme can be used in a high-performance fashion through it.
- JuliaFolds2 and Transducers could use some maintenance help (removing cruft / simplifying Transducers). It would be a good way to jump into the Julia ecosystem and provide volunteer service – do not hesitate to reach out to the organization.
- Not recent, but I found it only recently, ReadableRegex.jl is wonderful if you want a more pleasant way to write and understand regex incantations.
- Dynare, the macroeconomics modeling tool now has an official Julia translation.
- WeatherReport.jl - a weather app in Julia.
- ACAFact.jl - fast low-rank approximation of matrices with greedy pivoting
- Showflurry.jl - a quantum circuit SDK for the Anyon Systems Inc hardware.
- Malt.jl - an alternative for Distributed.jl used internally by Pluto.jl
- MusicTheory.jl - for representing musical objects (e.g. pitches and intervals) in Julia
- Optim.jl, one of the earlier function optimization toolkits in Julia is seeing quite a bit of polish in a new release.
- ComplexityMeasures.jl is a comprehensive package for… well, complexity measures. Large families of tools to estimate “complexity” (broadly defined) of timeseries and other data.
- MapMaths.jl and MapMakie.jl are some new tools for cartography. The linked thread briefly mentions already existing alternatives / related packages like Geodesy.jl and OSMMakie.jl and GeoMakie.jl and how the new packages related to them.
- SatelliteAnalysis.jl is now v0.3.0 with significant improvements and changes.
Soapboxes (blogs/talks):
- Consider subscribing to the French community newsletter (much of the shared materials are in English).
- Consider subscribing to the community calendar to be informed of upcoming virtual meetings and talks.
- Consider attending the triage meetings of the julia core contributors (organized on slack) – being a fly on the wall can be a great way to learn the nitty-gritty details of current priorities and development work. These are organized on the triage channel in slack.
- A wonder blog, Scientific Coder, discussing the dev-vs-scientist dichotomy in scientific computing and the two-language problem.
Sundry:
- Enzyme will be running their own conference. Would be of interest to compiler folks and autodiff folks.
Please feel free to post below with your own interesting finds, or in-depth explanations, or questions about these developments.
If you would like to help with the draft for next month, please drop your short, well formatted, linked notes in this shared document. Some of it might survive by the time of posting.