Discussion on the future for performant programming in Julia

[jt1]

Edit for future readers, the respones of @ChrisRackauckas in this thread, together with the response of @tim.holy here (which I missed) gives a clear enough plan.
The following gives an explanation as to why roadmaps are difficult.

Why open-source timelines/roadmaps don't work

In my time working on Julia, I’ve seen a lot of requests for estimates of when some feature will be finished, or a roadmap for how it’s going to be accomplished. This post is intended to summarize why satisfying these requests is sometimes difficult or impossible, and address the much smaller issue of how such requests can occasionally come across as seeming entitled or ungrateful. Over time I’ve tried answering these requests in different ways, trying out different approaches for making the poi…

TLDR: Look at the bold stuff.

I hope to have some discussion here possibly with members from the Julia Core team of what the future will bring for Julia. In my opinion, there is a lack of transparency after concerns raised in for example the mojo thread. I believe a timeline of features to tackle current problems would help the community much.

The main competitors of Julia right now are the deep learning DSL’s like Pytorch which are often used outside of deep learning just because they easily give performant code. IMO, Why are scientific programmers not using Julia instead?

I think that Julia only solves the 2 language problem, for those few that know how to write “mini-compilers” for memory management of arrays.
In the prime example of high performance computing, DifferentialEquations.jl, all required memory is cached beforehand like here. Once differentiation is required, it requires 400 lines of code just to get forward-mode AD with memory management to work efficiently. Not to mention that they had to develop a new datatype.

This approach takes too much time, is too hard for most developers and doesn’t scale to more heavy array calculations like in Machine learning. I would love to know if Julia will tackle this and how and when.

The difference with Pytorch which doesn’t have these problems is that Julia arrays don’t know their shape at compile-time and hence automatic memory management of arrays isn’t possible. I think having no separation between lists and arrays in Julia is a mistake, they should share a supertype though. This would be solved by adding arrays with known shape to Base and writing automatic array memory management for them and having AD work well with them. I Think MArrays from StaticArrays.jl is a good starting point. But with a name like ShapedArrays for example.

With this, it would be possible to statically compile most type stable code without shipping a GC, since most type stable functions only allocate due to Arrays. Moreover, compilation to MLIR, GPU and others should be doable.

I really believe that this approach is the right one for Julia since it is doable and since the requirement of having known array sizes in functions is fair for static compilation. I know I come over as entitled but I really love the work that has been going on in Julia, and I would love for Julia to become even more widespread. Am I too entitled to ask this? What is the opinion on this by the community?

[Eben60]

@jt1

The main competitors of Julia right now are the deep learning DSL’s like Pytorch which are often used outside of deep learning just because they easily give performant code. IMO, Why are scientific programmers not using Julia instead?

As an aside, I’d like to note there may be huge difference in understanding of what is considered to be Science (and more so, the meanings of the corresponding word are different in different languages), and thus, who is a “scientific programmer”.

I personally count Mathematics as meta-science, and Physics, Chemistry and parts of modern Biology as Sciences, everything else being “Humanities”, however mathematized

[tim.holy]

@jt1

In my opinion, there is a lack of transparency after concerns raised in for example the mojo thread. I believe a timeline of features to tackle current problems would help the community much.

I didn’t read that whole thread (it’s huge), so I’m not sure I understand your concerns. But just to make sure we understand what you’re asking for, do you think there’s a “lack of transparency” with respect to the timeline for the next great leap forward in battery technology? About when we can expect a cure for glioblastoma? How about the date on which we should be stocking up on food in preparation for the next pandemic?

[ChrisRackauckas]

@jt1

Once differentiation is required, it requires 400 lines of code just to get forward-mode AD with memory management to work efficiently.

Nope. Correction, that’s for forward sensitivity analysis, which is equivalent to forward-mode AD (with a few differences in computation order). Forward-mode AD of the solver works without any code from SciMLSensitivity, which is why it’s performed even if that package isn’t loaded.

@jt1

In the prime example of high performance computing, DifferentialEquations.jl, all required memory is cached beforehand like here

It’s very common in HPC to have strong memory bounds and have to know the exact memory requirement. This way we can enforce strong real-time guarantees on the amount of memory that is required by the methods. You can see this in action in the low-storage RK methods which actually test this property to ensure we have exactly the theoretical number of arrays allocated.

SciML/OrdinaryDiffEq.jl/blob/b7bb73f46669e6f35c33baf2b4a1a1dc9892a091/test/algconvergence/ode_low_storage_rk_tests.jl#L109

Of course, DifferentialEquations.jl is used in a lot of real-time applications where guaranteeing all memory is allocated up front to have an essentially static stepper is a requirement. So even with improvements, DiffEq has some applications which PyTorch’s model cannot support which we want to continue to support, and even support better (going all the way to static compilation and edge computing). Therefore it’s not going to go away as this code is more similar to C++ or Rust in terms of the application’s goals.

That said.

@jt1

This approach takes too much time, is too hard for most developers and doesn’t scale to more heavy array calculations like in Machine learning. I would love to know if Julia will tackle this and how and when.

The difference with Pytorch which doesn’t have these problems is that Julia arrays don’t know their shape at compile-time and hence automatic memory management of arrays isn’t possible. I think having no separation between lists and arrays in Julia is a mistake, they should share a supertype though. This would be solved by adding arrays with known shape to Base and writing automatic array memory management for them and having AD work well with them. I Think MArrays from StaticArrays.jl is a good starting point. But with a name like ShapedArrays for example.

With this, it would be possible to statically compile most type stable code without shipping a GC, since most type stable functions only allocate due to Arrays. Moreover, compilation to MLIR, GPU and others should be doable.

This is all being discussed and there’s a small PRs in motion. v1.9 and v1.10 had a large focus on TTFX and usability issues, with the end of v1.10 getting some things like improved stacktraces a major push (for example, improved stacktraces for automatic differentiation Frame registration for foreign JITs · Issue #49056 · JuliaLang/julia · GitHub and generally truncating stacktraces https://twitter.com/ChrisRackauckas/status/1661014235466563591). With those kinds of usability improvements and loading improvements, v1.10 is set to branch off its release rather soon.

With those changes in hand, we have been having many discussions about improving the way that high performance and statically compilable code can be written. This includes steps like:

• Improving the performance of the allocator (memory reuse between iterations · Issue #49881 · JuliaLang/julia · GitHub and Allocating empty array is 4x slower than Java · Issue #45160 · JuliaLang/julia · GitHub are example MWEs)
• Making the GC format faster to free
• Improving escape analysis and shape inference so that the compiler can optimize with known lifetimes
• Changing the implementation of arrays to be based on a generic memory buffer, which other types can then build off of.

The MWEs came from a recent priorities meeting so while it’s hard to put a timeline on things, I can at least say that the improvement of memory management on performance has now hit the top of the stack in terms of some JuliaHub priorities and I would expect a major focus in this area, along with static compilation, in the near future.

We will (hopefully) be writing a Julep in the very near future that details some motivations and some planned routes to solve some issues along these lines, so I’ll just cut here for now.

[jt1]

Thank you for the clear and well written plan @ChrisRackauckas, This is what I was looking for and i’m happy for the future of Julia and convinced me to keep using Julia.

Meanwhile I will try to program those ShapedArrays with memory management for a static subpart of Julia myself and if it works I might be able to help others with it.

[martin.d.maas]

@ChrisRackauckas

Improving escape analysis and shape inference so that the compiler can optimize with known lifetimes

Hey, that’s cool to know!

It’s currently beyond my skillset, but I think I would like to try to contribute to a problem like “shape inference”, so whenever there is something worth sharing (like an experimental package, or a writeup or whatever) please do.