A monthly newsletter, mostly on julia internals, digestible for casual observers. A biased, incomplete, editorialized list of what I found interesting this month, with contributions from the community.
“Internals” Fora (Slack/Zulip/Discourse/Github):
- It is always surprising how straightforward it is to modify the Julia compiler from within Julia. This discourse thread showcases how to implement Common Subexpression Elimination (CSE) at the julia intermediary representation level (IR) before it is sent to LLVM for compilation. LLVM of course has its own optimizations, including CSE, but there is value to doing some optimizations within Julia, where one has access to much more semantic information.
- A slack thread on what “interfaces” can look like in Julia, with interesting discussion of how multiple dispatch complicates things and examples of languages with multiple dispatch and multiple inheritance at the same time (Julia does not have multiple inheritanc yet).
Core Julia Repos:
- Dynamically Scoped Variables! This really came out of left field. We have been taught in school how terrible dynamically scoped variables are, just barely any better than global scope for everything. However, in recent year they have seen a resurgence in some very specific settings (e.g. multi threading, logging, configuration passing, request servers). If you are interested, you can learn more about it from the upcoming implementation in base Julia, as well as from the previous attempt at such a capability.
- Precompiled caches will soon be relocatable, a first step in making distributing precompiled projects (no waiting for compilation locally), and a valuable quality-of-life improvement for HPC cluster environments.
- Many improvements to the “effect analysis” that julia uses to prove properties of various methods necessary to permit more aggressive optimizations (e.g. proving a function is pure which might permit constant propagation or common subexpression elimination). Much of that information is also then sent to LLVM so that it can be used when optimizing the generated machine code. There was some recent discourse discussion around this as well.
- Along those lines, there is work underway to feed back information from the optimizer into the known effects, enabling yet more optimizations.
- A terminfo parser implemented in pure julia was merged, removing some external dependencies, and, more importantly, enabling future improvements to “fancy” REPL functionality.
- One such “fancy” functionality is the upcoming “Styled strings” work, which will enable much richer terminal output and a much easier API for styling such output.
- Cthulhu is a piece of dark magic that lets you investigate how angry the Julia compiler is with your code. The Julia vscode plugin is now capable of automatically visualizing this information for the user, warning them about type-unstable functions and more.
- The
--projectflag can now be set relative to the folder of the file being executed (instead as an absolute path or a path relative to “present working directory”) . Valuable when you want to execute a script with a particular Project.toml without having to specify both. - Improving documentation of how low-level math functions deal with domain errors,
missing, and other special cases. In spirit, a part of the slow process of documenting behaviors that are unsurprising for numerical programming folks but not obvious for newcomers. E.g. consider the discussion about julia’s attitude to aliasing or the discussion on integer overflow rules from last month. - Stacktraces and method errors with deeply nested parameterized types are now much easier to read in the REPL
- We might end up with a
publickeyword (new syntax) to help in better clarifying what is public and private API. This has been becoming more and more pressing given how Julia’s semver conventions are somewhat informal. There is some interesting discussion in that thread about possible drawbacks to the given approach and tradeoffs to consider. - Lazy loading of the standard library JLLs (binary shared libraries) leading to nice improvements in startup latency.
- Enhancements to defining primitive types with more sophisticated LLVM features.
- The julia runtime has a vast array of profiling endpoints, used by more and more external profiling/timing tools. There is a proposal for standartising a timing API for the low-level runtime.
Dustbin of History:
- Each function in julia has its own type, which is wonderful when we want type-stable higher-order functions, but it can lead to overspecialization (compiling way to many different methods for the same function). If you dabble much in functional programming, you would probably be interested in the solution provided by FunctionWrappers.jl and the very ancient issue about having this capability be in base julia.
“Ecosystem” Fora, Maintenance, and Colab Promises (Slack/Zulip/Discourse/Github):
- Satisfiability.jl is an interface package for a multitude of theorem prover tools, of great value to anyone that needs state of the art SAT solvers
- A very impressive and powerful (especially given how short the implementation is) eikonal and shortest path solver in Eikonal.jl.
- Mousetrap.jl is a quite wonderful new GUI-building tool providing a neater API around the intimidating GTK.
- With PlutoVSCodeDebugger you can now run your pluto notebooks through the debugger available in vscode.
- A wonderful R-like tidyverse-like tool for working with dataframes recently reached its v1.0. Check out Tidier.jl
- In the same family, check out TidierVest for web scraping.
- The JuliaPlots project is now able to collect funds through OpenCollective.
Soapboxes (blogs/talks):
- Consider subscribint to the French community newsletter (much of the shared materials are in English)
- Consider adding subscribing to the community calendar to be informed of upcoming virtual meetings and talks.
Please feel free to post below with your own interesting finds, or in-depth explanations, or questions about these developments.
If you would like to help with the draft for next month, please drop your short, well formatted, linked notes in this shared document. Some of it might survive by the time of posting.
