This month in Julia world - 2024-06

Community Newsletter
1

A monthly newsletter, mostly on julia internals, digestible for casual observers. A biased, incomplete, editorialized list of what I found interesting this month, with contributions from the community.

“Internals” Fora and Core Repos (Slack/Zulip/Discourse/Github):

  • There are considerations of how to remove some of Julia’s warts without a breaking Julia 2.0 release: a “strict mode” and versioning of export statements seem like a pretty big step in that direction.
  • The compiler now has a much neater “engine” to track (potentially multithreaded) inference and code generation.
  • Much neater autocompletion of emojis on the REPL.
  • This is a bit of a prolonged edge case frustration at this point, but progress is made again on fixing the Tuple{Union{}} breaking change edge case from 1.10. Discussed previously in the March issue of this newsletter.
  • Progress on the “wordage for bindings, not just function methods” discussed in the last month’s issue. Refactoring and cleanup being done for lowering of consts. Some related performance improvements to the use of global variabls.
  • Julia now has a more reliable mechanism to communicate breaks to internal unexported nonpublic functionality, hopefully leading to more stable packages that (ab)use unstable implementation details (e.g. JET, Revise, Cthulhu).
  • Improvements to the printing of stack overflow traces related to recursion. Some related PRs.
  • @atomic might soon permit indexing.
  • Improvements to tuple sorting.
  • Interesting experimentation with new recursive implementations for fold, map, setindex functionality for tuples, related to better loop unrolling capabilities in the compiler and to better type stability. [1] [2] [3] [4]
  • An interesting discussion on the interaction between convert and == and other equality checks on slack and a related github discussion on the type-safety of equality checks.
  • JET now has an official badge you can put in your readmes (together with your Aqua badge).
  • On master you can start julia with --trace-compile=stderr --trace-compile-timing to figure out how long compilation/precompilation takes for each methods.
  • A big step forward for the project to plug external garbage collectors into julia. Still, just refactoring for now.
  • The standard UUID library now has a UUID 7 implementation.
  • More detailed documentation of the new memory backends on which vectors are implemented.
  • An interesting example of using @constprop annotation to improve the type stability of a function.

Dustbin of History:

  • Unicode subscripts and superscripts are wonderful when writing more legible code, but frustratingly a few random latin letters are not available… Some time ago the Julia community started preparing a proposal to the unicode consortium to remedy that situation, but it has lately lost steam (any volunteers?).

In search of contributors and new maintainers (specify novice/moderate/expert and internals/domain background necessary):

Ecosystem Fora, Maintenance, and Colab Promises (Slack/Zulip/Discourse/Github):

  • GTPSA is a new library for Taylor-style autodiff, with a new Julia interface.
  • Julia was used to prototype the new audio codec in WhatsApp.
  • Significant improvements to linear algebra on multinode computing clusters with Dagger.jl, discussed on slack.
  • AlgebraOfGraphics.jl plotting library is getting a healthy refresh, discussed on slack.
  • A slack discussion of improvements in DifferentiationInterface.jl – this package is becoming a valuable cornerstone for the autodiff efforts in Julia.
  • Some recent publicity about Julia in HPC, discussed on slack.

Notes from other ecosystems:

  • NumPy 2.0 breaking release is out. Discussed on hacker news.

Soapboxes (blogs/talks):

  • Consider subscribing to the French community newsletter (much of the shared materials are in English).
  • Consider subscribing to the community calendar to be informed of upcoming virtual meetings and talks.
  • Consider attending the triage meetings of the julia core contributors (organized on slack) – being a fly on the wall can be a great way to learn the nitty-gritty details of current priorities and development work. These are organized on the triage channel in slack. Minutes are kept.
  • Julia tshirts made with Luxor.jl

Sundry:

  • A “Julia for HPC” training course will run in Germany in September.

Please feel free to post below with your own interesting finds, or in-depth explanations, or questions about these developments.

If you would like to help with the draft for next month, please drop your short, well formatted, linked notes in this shared document. Some of it might survive by the time of posting.

59 Likes
2

Thanks for the attention! I’ll probably close these PRs as I’m working on a new design, but probably it’ll be a package for some time before I try to upstream it.

The idea with the open PRs was to keep, together with a tuple, a singleton type representing the length of a view into the tuple.

The idea that I’m currently experimenting with is to create a new recursive data structure for a constant-size heterogeneous collection. The first thing I tried was simply something like this:

struct HeterogeneousList{ElementType,Rest}
    first_element::ElementType

    # When `rest` isn't a `HeterogeneousList`, the object represents an empty collection
    rest::Rest
end

A problem with the above design was that there’s no way to describe, e.g., a list of indeterminate length, all of whose elements are Int (Tuple{Vararg{Int}} in tuple parlance), or a list whose first element is Bool while any others are Int (Tuple{Bool,Vararg{Int}} in tuple parlance).

The current design looks like this:

  1. There’s a homogeneous list type (representing a constant-length homogeneous collection, so all elements are of the same type)
  2. Then the heterogeneous list type is implemented as a list of “homogeneous extents”, each of which is a homogeneous list. The idea is for each homogeneous extent to be maximal

This design is actually supposed to allow more descriptive power than the combination of Tuple and Vararg, as it should allow describing types such as “an indeterminate number of Ints, terminated by a Bool”. This is something that’s not possible to describe as a tuple type in Julia’s type system. In the end this should allow for improved inference, especially with abstract argument types. For example, currently an operation such as reverse loses information when given a tuple type with Vararg:

julia> f(t) = reverse(reverse(t))
f (generic function with 1 method)

julia> Core.Compiler.return_type(f, Tuple{Tuple{Int,Vararg{Bool}}})
Tuple

Ideally, the inferred type would be Tuple{Int,Vararg{Bool}} instead of just Tuple, and my design above should allow for that.

In practice my package would probably be used like so:

  1. convert a tuple to a heterogeneous list
  2. apply the required operations on the heterogeneous list instead of on the tuple
  3. convert the result back to Tuple
6 Likes