This month in Julia world - 2024-11

Community Newsletter
1

A monthly newsletter, mostly on julia internals, digestible for casual observers. A biased, incomplete, editorialized list of what I found interesting this month, with contributions from the community.

“Internals” Fora and Core Repos (Slack/Zulip/Discourse/Github):

  • Making the creation of Memory an “intrinsic” – a special-cased lower-level operation for which the compiler can know more, thus providing better optimisation. Maybe soon enough small arrays will be allocated on the stack and StaticArrays.jl will be needed less. Original PR and the PR being split in more digestible pieces.

  • The Julia compiler is now just a normal (standard) library that you can install and develop independently of Julia, greatly improving the ergonomics of development. See the main PR and discussion on slack.

  • Quite a bit of progress on “compiler plugins”: Few months ago Valentin prepared a draft of how code analyzed by external “abstract interpreters” can be executed within Julia (tools like Enzyme/GPUCompiler/Cassette use various cumbersome workarounds for calling externally analyzed/maniputed/compiled Julia code back into the Julia runtime). He gave a talk about it at JuliaCon. Recently Keno suggested related capabilities stemming from his work on making the compiler just another “independent” standard library. After further considerations, a minimal set of capabilities was distilled into a small PR that permits invoke to accept CodeInstance. invoke is the standard way to hook into the method lookup capabilities – you give it the function you want to call, and the type signature for the method you want to call – thus it provides a neat way to override which method is being called. Now it can also be used to call into code compiled by independent Julia compiler plugins, with different semantics compared to the main compiler (e.g. during development of new compiler versions, or completely independent autodiff or GPU compilers). Check the new docstring for details.

  • The “pretty printing” infrastructure in Julia is sophisticated but still quite underdocumented. Topics about it have come up often. One new summary was discussed on slack.

  • This newsletter announced the transition to LLVM 16 only last year. We already are on our way to use LLVM 19, so 16 is going out.

  • Fun edge cases with parsing function()() end discussed on slack.

  • Standardizing how and when global = gets lowered, sticking to using setglobal!, leading to fewer special cases in the compiler passes.

  • disable_new_worlds() can now be called, freezing the world age, stopping invalidation tracking, significantly improving the performance of code loading, as long as you know that the new code does not invalidate anything. At the same time we also got a lot more standardization and cleanup to when world ages actually get incremented.

  • Maybe nth will be added to Base to generalize first and last.

  • LinearAlgebra was moved to its own repository (but it is not yet an external independently-versioned standard library)

  • The base ReentrantLock might get an optional warning message on detecting lock contention.

  • A potentially beneficial new coupling between inference and optimization, where inference can benefit from type information revealed by optimization.

  • An interesting discussion and potential changes to promote_rule and promote_type and when to use which. Stemming from an earlier conversation on related antipatterns.

  • The Julia 1.12 feature freeze is announced – January 8th.

  • An informative discussion on the lack of non-blocking IO in Julia and what it would take to implement it.

  • Proposal for significant improvements to Julia’s IO interface. Direct link to the proposal itself.

  • A dive into the details of constant propagation and when can the compiler figure those out, with a simple example.

  • Yet another insightful discussion on Julia’s type system: nominal typing vs structural typing/traits.

In search of contributors and new maintainers (specify novice/moderate/expert and internals/domain background necessary):

Ecosystem Fora, Maintenance, and Colab Promises (Slack/Zulip/Discourse/Github):

Julia Autodiff ecosystem (enough topics for its own category this month):

  • For floats 0.0 !== -0.0. The various types of equality checks have come up often on various community forums, but this time it is about an exciting interaction between this surprising float inequality and autodiff. Discussed on slack with a reference to this informative manuscript.

  • Mixed-mode sparse Jacobians at insane speeds available in DifferentiationInterface.jl, discussed on slack

  • How about finite-differences mode in Enzyme, discussed on slack

Notes from other ecosystems:

Soapboxes:

See also: French community newsletter, community calendar, minutes of triage meetings

Please feel free to post below with your own interesting finds, or in-depth explanations, or questions about these developments.

If you would like to help with the draft for next month, please drop your short, well formatted, linked notes in this shared document. Some of it might survive by the time of posting

44 Likes
2

s/word/world

1 Like
3

s/constraint/constant/

1 Like
4

You meant either !isequal(0.0, -0.0) or 0.0 !== -0.0 instead.

2 Likes
5

I am very excited about Oscar.jl’s experimental PR: Clifford algebras (clifford_algebra) and Clifford orders (clifford_order)—for quadratic spaces (quadratic_space) and lattices (lattice).

2 Likes
6

Sounds exciting, would love to see a few lines of example usage!