A monthly newsletter, mostly on julia internals, digestible for casual observers. A biased, incomplete, editorialized list of what I found interesting this month, with contributions from the community.
“Internals” Fora and Core Repos (Slack/Zulip/Discourse/Github):
- A wonderful essay on microbenchmarks and volunteer labor and community by Jakob.
- Julia now has a generic memory type, representing simply a reserved buffer space in the computer’s memory which can then be used to implement higher level containers like Array. One of the big wins here is that Array is now fully implemented in Julia, making it much simpler. Moreover, the compiler now can reason about and optimize Array operations, because they are not implemented in opaque C code. See the initial design document, the pull request, and this slack thread.
- A long slack discussion on allocating and non-allocating inplace sort implementations, in particular noting that allocating sort implementations can be faster than the nonalocating ones.
- In Julia’s type system, the type of types can be quite confusing. Recently popped up in this slack discussion, but consult also these very old issues and discussions: [redesign plans], [tuples of types dispatch confusion]
- Is
@assertmisused in the Julia ecosystem, is it supposed to mean “debug-time checks that can be turned off” or “data validation checks that are always on”? - A discussion of the subtleties and edge cases of what is reasonably called “type piracy”
- What can potentially be done to make Julia’s garbage collection more “realtime friendly”.
- A slack discussion on the unfortunate issues in the way the trait API of iterables is designed and potential future improvements.
- A discourse thread on workarounds for when your CPU is modified by an update meant to protect you from sidechannel attacks (at the price of much worse performance).
Core Julia Repos:
- Precompile files (i.e. the cache files containing the compiled machine code, usually stored in a hidden .julia folder, and which are the recent solution to the TTFX problem) are now relocatable. This is a prerequisite for future endeavors like distributing precompiled files so that users do not need to compile installed libraries themselves.
- There has been work for a while to support alternative Garbage Collectors in Julia. An official interface for hooking up such alternatives is now in a draft state.
- libffi is a library that lets you call functions “compiled by someone else” in runtime, without needing a compiler/linker. It is particularly useful for interpreters that might want to call into a C library without access to a compiler. This draft PR explores using libffi in compiler-less sessions of the Julia interpreter for ccall support.
- AnnotatedStrings is now merged!!! It was a grueling endeavor, but thanks to @tecosaur we now have support for wonderfully richly formatted strings in base julia (in the terminal and beyond).
- Julia is transitioning to LLVM 16. The last step is still pending. Relatedly, LLVM 16 is now being build and made available in Yggdrasil, Julia’s repository of compiled external packages.
- An interesting new optimization in the compiler which might be a good spot for newbies to learn more about how optimizations are written in Julia itself. This new one simplifies try-catch blocks if it can prove that no exceptions can be triggered.
Ecosystem Fora, Maintenance, and Colab Promises (Slack/Zulip/Discourse/Github):
- StatisticalMeasures.jl was recently released, a package providing a wide variety of measures that can be calculated for probability distributions (e.g. entropies, mean errors, etc). Would be of use to anyone doing ML or probability programming or Bayesian “stuff”.
- Conditions.jl provides a very interesting alternative to exception-based control flow. It implements the Common Lisp “signaling a condition” approach instead of the typical “throwing an exception”.
- AbstractDifferentiation.jl has a new more polished stable version 0.6. This package is of great use to people that want to ensure their code works with many different differentiation frameworks.
- Finch.jl is a fantastic “nested loop optimizer” that supports sparse and dense multidimensional arrays. It evolved out of the older TACO project.
- The standard libraries shipped in Julia are becoming more decoupled from Julia itself, with a ton of logistics benefits in terms of faster bugfixes, faster development, easier testing, and more. That would require some minor changes in how people declare compat bounds in newly released versions of their packages. The Compat bot on github mostly automated that work.
- Consider adding the “typos” CI action to your repository – it was recently added to Documenter.jl and it catches a lot of annoying spelling errors with virtually no false positives.
- Quantica.jl is a wonderful new tool for modeling crystals and other solid state structures.
- We discussed Mousetrap.jl recently - a tool for building GTK GUIs. It is a wonderful project that has gained many new features in a short time, including support for embedding GLMakie.
- OptimalSortingNetworks.jl is a ridiculously fast tool for sorting small containers, in particularly typestable sort of tuples.
Soapboxes (blogs/talks):
- Consider subscribing to the French community newsletter (much of the shared materials are in English)
- Consider subscribing to the community calendar to be informed of upcoming virtual meetings and talks.
- There is an irregular “Julia Gender Inclusive Community Hour”, frequently with very interesting technical presentations. Consider clicking “watching” on these Discourse threads for future announcements.
Sundry:
- Ousterhout’s dichotomy is a useful to know principle that Julia proves to be a fallacy. It distinguishes scripting languages from systems languages, i.e. another name for the two-language problem.
Please feel free to post below with your own interesting finds, or in-depth explanations, or questions about these developments.
If you would like to help with the draft for next month, please drop your short, well formatted, linked notes in this shared document. Some of it might survive by the time of posting.