Obviously, we all know this question has been asked numerous times.
I am using latest Julia 1.8, and in my current research I am using Julia Makie/AlgebraOfGraphics for most of the analytics and plotting, and I am encountering a lot of usability issues due to the TTFP problem. I know Julia has done a lot of optimization, but I am easily require 2-3 minutes before all packages are loaded and stuff start to draw. Every time I call a new drawing function, Julia takes minutes to compile stuff from the packages (I assume Makie), even though they are supposed to be precompiled.
You might say I should just keep a Julia instance alive, but this leaves me a lot of dead variables during the exploration phase. Since Julia cannot âclean workspaceâ, I have to restart the instance every now and then, which brings back the loooong start up time.
An alternative I use is Pluto. This works fine, but I often have to run multiple instances of it, and it gets impractical when the document goes really long.
Lastly, after the âexplorationâ phase, I need to put all code into scripts so I can reproduce publication-quality figures as easily as possible. And this, takes even more time because scripts are run cold.
I just cannot find a way to cut down this latency. Am I configuring my system incorrectly or something?
I donât think this is the right answer. What OP is describing really shouldnât be happening even without a sysimage. If Iâm reading it right, Op is saying that precompilation isnât being saved for some reason.
I donât know if precompile is actually working. It might be⌠because I donât know if 2-3 minutes loading is normal for precompiled packages. In any case, it is a lot of time.
Sysimage is quite complicated, and I would like to avoid it when I donât know what packages I need (during exploration, for instance)
I have a fairly different experience even without a sysimage, so it would be important to check some basics.
First, do you consistently use a Project.toml file? A typical workflow in julia would be to
start a project in some preferred folder (e.g. cd my_project; julia --project=. where . means âcurrent folderâ)
then install the necessary packages, e.g. ] add DataFrames CairoMakie DifferentialEquations
these things will precompile as they are installing; that can take multiple minutes, but it is done only once
then whenever you want to work on the project you start julia with the same project flag and do using CairoMakie; plot(...); the import takes about 10 seconds (which is indeed much too slow) and the first plot takes 20ish seconds (also too slow); plotting after that is extremely fast
Do you have that baseline? If things are worse than this, there is some important problem to be fixed. In particular, make sure you have a Project.toml setup you are happy with so you do not need to install extra libraries every time you do some more exploration. Library install and precompilation should ideally be one-time task at the start of a project.
Then, if the 10 second import for CairoMakie is still frustratingly slow (it is), you have a couple of options:
Reach a zen state of mind and accept it because it is a reasonable price to pay for the multimethods-with-compilation paradigm of Julia (that is a joke â while there are interesting theoretical reasons for why it is so difficult to cache code, this community is solving these hard problems one after the other)
Wait for improvements by Tim, Valentin, and Jameson to land in Julia and mostly fix this over the next couple of julia releases
Make a custom sysimage for yourself with just a couple of clicks (and a bit of compilation wait) in VS code
Reading through the post you made as I was writing this, you are saying that the exploration part of your work involves trying out new packages. I do not think julia supports that workflow well, because a lot of stuff needs to be compiled each time you install new package (this is drastically slower than just importing already installed package, which is what most of my post is about). I think today the only solution to your problem is please try to install most of the packages you care about in one go at the start of your project. This might get better in the future, but I do not think it is a priority for the community. Basically, what you are asking for seems to be faster installation of packages (because you update what is installed in your project often), not necessarily faster imports (that might be annoying, but not the root cause of the slowdown).
I do not use projects, but have a bunch of loose scripts running under global environment. The import of CairoMakie alone does take 10-30 secs, but since I use quite a few other packages, it is slower. The time you report is not that far from my experience.
I just did a sysimage. It takes about 6min, but I will see how it works.
I donât really care much about installation speed. I was referring to the issue which I need to rebuild a sysimage every time I update/change/add a package.
I do suspect you will have much better experience using projects. That is true not just for Julia, but for Python, Rust, and R and many other languages. The reasons span from organization/logistics/reproducibility/sharing/logistics to âsimply fewer things move around when you update a minor libraryâ.
If you update any of the many many indirect dependencies of Makie through adding or updating a different package that shares a dependency, at the next time you load Makie it will precompile again. The more packages interact with each other this way, the more this happens. So keeping projects small and updating only once in a while keeps this annoyance smaller.
You need to start using projects, and I recommend you delete your entire global environment and start over.
rm ~/.julia/environments/*
Julia will create a new global environment when you run it the first time. I have exactly 3 packages in my global environment:
(@v1.8) pkg> st
Status `/var/local/dlakelan/dotjulia/environments/v1.8/Project.toml`
â [ac637c84] AbbreviatedStackTraces v0.1.4
[14b8a8f1] PkgTemplates v0.7.29
[295af30f] Revise v3.4.0
Info Packages marked with â have new versions available
I could imagine more, but not much more than 10. It really should just be packages you rely on for fundamental things that you do outside local environments.
To create an environment for your project, just start Julia in the directory youâll be doing your work in, and do
] activate .
] add ...
After activating the current directory as the current environment, any packages you add will immediately appear in the local Project.toml and Manifest.toml
Whenever you are working on a project, just activate the current directory before running your code. This is the only clean way to get Julia package management to work on your behalf.
Using projects is good, but I do not think this is a good advice.
After back and forth experimentation, currently I really donât care that my global environment is bloated of packages. That is actually useful if one wants to use some package for a tiny test without adding it to the current project, and does not affect the packages that belong to the project itself. Other packages that are useful to have in a shared environment (and I never use any other than the main one): JET, Chutchlu, LiveServer, BenchmarkTools, UnicodePlots, OhMyREPL and dependencies, etc.
True, but once one gets used to environments and has a clear picture of what packages actually belong to the project in case, having the possibility of using other packages is very handy during development.