About reproducibility of Julia program run times

[uwestoehr]

I have a general issue with Julia. I know there is no solution but something should be done because as it is collaborative work is hard:

The runtime of Julia programs is not stable. I don’t see fluctuations in the range of 10% but of 100%.

StaticArrays is a good example why I struggle with Julia (I come from C++ and also Lazarus).

Take for example this code:

matrix = rand(Float64, 1500, 2)
@elapsed sDataMatrix= SMatrix{1500, 2}(matrix)

This takes here 3.22 seconds.
But only at the first run. When I repeat this, it takes only 1.8e-5 seconds.

This is what I encounter all the time. My programs run at first super slow. Then I just run them again, they are very fast. For example the first plot a program makes can take up to a minute, later it is just a fraction of a second. Or the loading of a CSV file, or the generation of a tensor etc.

Then I often encounter that Julia is kind of becoming slow. When I restart it and run my program again, I measure 10% faster run times.

Or I have Julia running and run @elapsed on the same function 10 times in a row. The first run is super slow as always, then Julia becomes faster with every run. Then I run another function and subsequently the function again, it is 10, 20 or even 40 % slower than it was at its last run.

For me the bad thing is that as a physicist and engineer I really like Julia, also the plotting is great, but I must collaborate: I have to send my code to a colleague who has the same system as I have (latest Windows 10 64bit, 16 GB RAM), but he could get 50% faster runtime, or only 10 % of my runtime. Then he tries again and get different values. Or I restart Julia and everything is slow, then I restart it is faster and so on.

I am used to that I can send a program to others and the runtime they observe is stable for the and for me and also the runtime between theirs and my PC is stable. I don’t need to break speed records but need a reliable compiler that keeps run times stable.

Maybe I can compile my *.jl file in a specific way to improve the situation?

Note that I read

and also the advises in this thread

were very helpful.
but none of the improvements I achieved are stable. First after 10 subsequent runs of the same program, function or code in general, the measured @elapsed run times are stable to make code comparisons.

And comparisons across PCs are just impossible. I mean if another PC is 20 % faster with a C++ program then it is 20 % faster every day on the program code, reproducibly. But not with Julia.

[uwestoehr]

For a strange reason I was not allowed to link existing forum discussions and not even code like this:
https://docs.julialang.org/en/v1/manual/performance-tips/#kernel-functions
That is Julia’s own website.

I am also not allowed to edit my posts. Therefore I apologize for typos.

[uwestoehr]

Maybe I should mention that I came to Julia because I used and Python. Python faces the same problems but it is more a rapid-prototyping language, not designed to be performant (to get reliable code there is Cython etc.).
Julia markets itself as fast by design. In principle it is fast, but unless the speed is not reproducible it is not as useful as it could be.

Again, I really like Julia and for engineering laboratory applications it is great. But I cannot build code for customers or even for colleagues. Sure, it runs, but for example on evaluating real-time data from a sensor, run time is important.

[karei]

@uwestoehr

I am also not allowed to edit my posts.

You can click this button to edit your post,

and this one to create a hyperlink.

@uwestoehr

but need a reliable compiler that keeps run times stable.

Currently, you can use PackageCompiler.jl to compile your program into a binary file for stable execution times. Only the file size obtained by this method is very large (500MB~1GB).

Added: Julia is slow on the first run and fast on later runs. This is because of JIT compilation, but I’m not sure about the specifics. But you can measure the execution time from the second run to get stable data, preferably using BenchmarkTools.jl. From the second onwards, the time fluctuates mostly depending on garbage collection (as far as I know, C++ also fluctuates in time.) .

[mrufsvold]

The behavior you’re experiencing is Julia’s Just-Ahead-Of-Time compilation.

When you execute code that defines a new function, no compilation happens. The first time that function is called for a specific set of input types the Julia compiler kicks in and compiles the optimal machine code for that set of input types.

As of Julia v1.9, packages can define precompilation routines to trigger the compiler at package installation time which are then cached and can be used between sessions. However, user code is not cached between sessions, so each time you start a new Julia session, the compiler has to recompile all the methods for functions you define and methods for functions that are defined by packages but not hit during the precompilation routine.

This “slow” first execution issue is called “Time to first execution” (TTFX) or sometimes “time to first plot.” If you search the Discourse for these phrases, you will find LOTS of discussion.

There are two remedies.

1. PackageCompiler.jl as @karei mentioned. You will need to have some “dry run” code that will call all the methods you want precompiled. Warning – this produces a full Julia System image that is quite large.
2. This is not yet published, but in Julia v1.12, there will be a new ahead-of-time Julia compiler that will produce smaller executables.

[ufechner7]

Apart from using PackageCompiler.jl you can:

• write your code as package and include a @compile_workload section
• use Linux instead of Windows
• use a CPU that has only fast cores
• use a plotting library that is based on C++
• use a desktop PC instead of a Labtop

to improve the reproducibility of execution times.

The first point makes sure compilation happens when you install the package and not when you are using it the first time in a session.

Achieving reproducible timing on Windows is impossible because Windows is doing a lot of expensive things in the background like updating and virus scanning on which you have no influence.

[uwestoehr]

@karei

You can click this button to edit your post,

I don’t have this button. This is what I see:

And whenever a post of mine contains a URL, when In try to post, I get a message that I am not allowed to use URLs. This applies also when I want to cite a link another user posted.

It seems someone sets me at a list as if I would be a spammer.

[uwestoehr]

@mrufsvold

This “slow” first execution issue is called “Time to first execution” (TTFX) or sometimes “time to first plot.”

Thanks. I searched for it now. The point is that I would be able to live with a certain TTFX but it varies a lot. I mean I start a new REPL and execute just:

using CSV, DataFrames, Plots

This can take 10 seconds or 30 or 75. But alone the time until the REPL is up and running varies a lot.

I was asking here because in a laboratory the time of the workers is costly. If a person has to spend in total e.g. 5 minutes a day just to watch a program starting, I produced about 4 € costs. Now scale that up to several laboratories.

Maybe I was naive but I started to learn Julia with having in mind to be able one day to provide nice applications to visualize and evaluate real-time data of sensors. If for this a paid service (e.g. a commercial compiler) would be necessary this would not be a problem. But of course delivering a 1 GB package for a small program is something likely not accepted by customers.

[uwestoehr]

@ufechner7

• use Linux instead of Windows
• use a CPU that has only fast cores
• use a plotting library that is based on C++
• use a desktop PC instead of a Labtop

I cannot tell others what they have to use. For engineers: we all work with laptops because we have to switch between laboratories, offices or on-site visits.

I have never seen Linux at a desktop at any company. Personally I gave Julia a try. I use a Linux distribution called CachyOS and almost all programs run faster at CachyOS than on Win 11 on the same laptop. But my problem is not that things might be 10 % faster or slower, but that the speed is not stable.

While we are at general things, as a newbie I experimented with AI to help me with Julia and the result is that PerplexityAI works at the moment the best for me: I can upload my code and let it review it or I describe what I want to do and it writes me code that works always directly. The only drawback is that its code is often not the fastest possible code. I also tested DeepSeek, Le Chat and ChatGPT. The latter is also suitable but it provides too often code that does not run. The other 2 are not yet well trained for Julia in my opinion. However, they work great for Python.

[uwestoehr]

@uwestoehr

I don’t have this button. This is what I see:

I just got a message from the system that I was now granted the right to edit my posts. It seems as a newbie someone must allow me this.

[danielwe]

Take a close look at PrecompileTools.jl. To remove the startup latency for Julia code, the trick is to

• structure the code as an installable Julia package, and,
• use @compile_workload from PrecompileTools.jl to move compilation from startup time to package installation time.

As others mentioned, the next version of Julia, 1.12, will include tools for compiling Julia code down to a binary application. At the moment, the closest you can get is generating a sysimage or app using PackageCompiler.jl. You may or may not find that this serves your needs; I’d recommend exploring the PrecompileTools.jl route first.

@uwestoehr

I was now granted the right to edit my posts

There are several restrictions on newly registered users, mainly to limit the ability of spam bots to wreak havoc on the forum, I think. Maybe a moderator intervened on your behalf, but it could also just be that you reached the engagement threshold for earning these rights.

[abraemer]

@uwestoehr

Take for example this code:

matrix = rand(Float64, 1500, 2)
@elapsed sDataMatrix= SMatrix{1500, 2}(matrix)

Please note that using StaticArrays.jl is typically not a good idea once you have dimension larger than a couple hundred (IIRC). StaticArrays.jl puts quite some stress on the compiler and thus inflates compile times.

There are many many ways to write a Julia program depending on what your constraints are. It is possible to optimize programs also with respect to compile times/startup latency.

Instead of posting

@uwestoehr

I know there is no solution

you could try posting a small(ish) code snippet that reflects your problems. It is very likely that you will receive helpful comments here

Personally, I never experienced the amount of variation in startup times you describe in the last 4yrs. Neither on my work laptop nor on HPC clusters. Maybe there is just some misconception on you end. Anyhow, I am quite optimistic that there is a solution to that. Especially if you are new to Julia, there are some pitfalls that can cause suboptimal perfomance.

[Benny]

It’s hard to say what you’re running into without a runnable example, and it’s likely not all of it is non-deterministic. Some possible sources of non-deterministic runtimes:

• compiling things that weren’t compiled already. Note that running an expression for the first time doesn’t necessarily get all the compilation over with, and it’ll occur more with type instability and globally scoped code. Use @time instead of @elapsed to see those stats.
• heap allocations depend on the state of your heap, especially if you don’t let objects become garbage
• garbage collection
• heavy runtime dispatches, especially with poor type inference
• OS scheduling, and some OSs do a lot more in the background
• CPU throttling, e.g. controlling high temperature or conserving power

Some of those obviously apply to a C++ executable as well, so you might have ruled those out already.

I can also say that the previous comments about precompilation to handle compilation latency will help, but you should still expect lesser latency from loading precompiled code and running __init__s, which occurs when importing a package for the first time in a session. The more code a package precompiles, the more load latency; though this usually saves a lot of time compared to session-wise compilation, it has warranted smaller core packages being split from larger ones, like how NumPy is split from SciPy. Loading code isn’t avoidable anywhere, so the only way to reduce repeated load latency is to keep the session open as long as possible. If you reuse a small set of working global variables as inputs to methods, it’s very doable in many interactive languages.

[Eben60]

@uwestoehr

Again, I really like Julia and for engineering laboratory applications it is great. But I cannot build code for customers or even for colleagues. Sure, it runs, but for example on evaluating real-time data from a sensor, run time is important.

In addition to the specific suggestions you were given, some general opinion:

• Having (soft or even not-that-soft) real-time application with Julia is feasible, e.g. Case study: Real time hardware control for adaptive optics with Julia or Home · ReactiveToolkit.jl , however may require a lot of know-how.
• Sending code to colleagues can work if you provide them with detailed instructions and warn them about the pitfalls. I’ve successfully done that (GivEmXL.jl).
• For sending code to customers: IMO Julia is simply not there yet (but that is seen as an important target in the language development). What could currently probably be done is a some kind of a cloud solution.
• Your kind of (intended) Julia usage is (still) less common, meaning you may need to invest more-than-average time to learn suitable practices.
• I expect tools to generate redistributable Julia code to to get past experimental stage in a foreseeable future.

[goerz]

@uwestoehr

The point is that I would be able to live with a certain TTFX but it varies a lot. I mean I start a new REPL and execute just:

using CSV, DataFrames, Plots

This can take 10 seconds or 30 or 75. But alone the time until the REPL is up and running varies a lot.

Not withstanding the various tips people have given in this thread: This is completely expected. The variations depend on how much previously uncompiled code has to be compiled.

Julia is a compiled language, and compilation takes time. When Fortran was my primary language, it would be pretty common to wait half an hour for the entire library stack to compile. The difference with Julia is that the compilation is gradual: compilation happens the first time something is run. That makes it less predictable.

As others have said, you can make it a little bit more predictable with manual precompilation, giving you something that’s a little closer to “up-front” compiled languages like C/C++/Fortran, but I wouldn’t consider this standard usage.

Compiled languages always involve a tradeoff between compilation time and run time. Compared to, e.g., Python, you have to wait for the program to compile, but the result will run much much faster. In common scientific workflows, where you compile for a few minutes (once, until you change the program) and then have the program run for hours or days, that’s more than a worthwhile tradeoff.

That’s not to say that I’m not also sometimes slightly annoyed at waiting around for Julia to instantiate environments (which is also dominated by compilation: doing the pre-compilation of all the packages that are being installed). With every version of Julia, there’s typically some progress at improving the TTFX experience. But still, compilation has to happen somewhere, at some point, so there are limits to what is possible.

@uwestoehr

If a person has to spend in total e.g. 5 minutes a day just to watch a program starting, I produced about 4 € costs.

It sounds like you have a requirement for extreme development responsiveness (developers not waiting around for compilation). If that’s the case, that is, if your code runs in seconds and thus runtime is not a consideration, Julia may not be the most suitable language for your situation. I might recommend Python in that case. Although Python can also take considerable time to set up environments; alas, that has to be done less frequently.

[nilshg]

@goerz

With every version of Julia, there’s typically some progress at improving the TTFX experience

Not to derail this thread, but this trend has unfortunately been broken by 1.11 (and by the looks of it also 1.12)