I assume you’re new to Julia, this is the famous time to first plot problem. Julia takes basically as long for the first plot as it would to compile any other static plotting library from source. It sucks but it’s the current state. Afterwards, plotting is fast. In the future, there might be binary caching which could help a lot, but it’s still speculative.
I appreciate the responses. I am relatively new to Julia (about a year). I have been developing Mandelbrot set plots (pixel by pixel) using GtkReactive. I save them as png files and make movies (typically 4k to 8K frames). For a 2580x2000 pixel plot with up to 9000 iterations per pixel, time to first plot is typically a couple of seconds. (I have a fairly high end PC with 20 cores which are all used.) My program works great and is blindingly fast. All my interactions (zooming in/out, selecting an area, shifting the center are all using the mouse and/or keyboard.) However, I have been frustrated by my lack of understanding GUI development using GtkReactive. Makie looks like a much cleaner GUI development environment.
I was aware of but didn’t think this was a time to first plot problem. My normal way to run a script is to use the VSCode Run command. However, I tried using the including the script from the REPL, made a slight mod, and it was indeed almost instantaneous with this small test case. Doing the same this with the VSC run command takes the same long time each time.
Questions:
Why the big difference? using include from the REPL vise the VSC run command?
Why doesn’t my GtkReactive version have this problem?
Any thoughts about GtkReactive vice Makie?
It will take me some time to follow up on the other suggestions.
A big thanks to all, Harvey
As you describe it, seems that it is some problem with Make and VSCode, so while those advices will be useful in general, probably they won’t solve that specific problem.
Is PackageCompiler reliable though? I had the impression that it could introduce weird bugs because the sysimage can silently violate the version constraints on dependencies. See this part of the documentation:
It should be clearly stated that there are some drawbacks to using a custom sysimage, thereby sidestepping the standard Julia package precompilation system. The biggest drawback is that packages that are compiled into a sysimage (including their dependencies!) are “locked” to the version they were at when the sysimage was created. This means that no matter what package version you have installed in your current project, the one in the sysimage will take precedence. This can lead to bugs where you start with a project that needs a specific version of a package, but you have another one compiled into the sysimage.
I’d love to have a fix for Julia’s compilation latency but this sounds like it can introduce awful bugs… Basically you should not install any package after you made your sysimage. Or did I misunderstand this warning?
Would be good to know which exact command you’re using. Julia: Run File in New Process will incur the compile and start-up cost each time, whereas Julia: Execute active File in REPL should be similar to includeing your file into a long running Julia session.
Why doesn’t my GtkReactive version have this problem?
Because Gtk is a C library, and GtkReactive just has a little bit of Julia code around it.
Makie is almost 100% Julia, and currently relies on ~126 Julia packages to create plots, so in other words it’s a huge Julia code base.
My normal way to run a script is to use the VSCode Run command
I use VSCode interactively most of the time, almost never restarting the julia process.
I’m not sure if I’m misreading the VSCode docs, or if the interactive code execution isn’t documented very well, but it’s explained here at least:
Btw, we’re working right now on Julia & Makie improvements, to cut down the compilation latency a lot…
We may see those improvements end of the year / start of next year.
It’s not that bad. It takes a few minutes to compile the sysimage (maybe a half hour if it’s a very big code base). When packages are upgraded, just make a new sysimage. Pay the compilation time up front while you make a cup of tea. I guess the worst thing would be if you want to use new packages that need OLD versions. Fortunately this is rarely the case.
I’ve tested the code here, and it is a “time-to-first-plot” kind of issue. If you just include it twice in the same Julia section it is fast:
julia> @time include("./makie.jl")
Starting Program
Initialize the figure
del_t = 4.155311822891235
Display the figure
del_t = 6.562004089355469
Get the figure Axis
del_t = 52.51437592506409
Plot …
del_t = 0.3330061435699463
Leaving program
70.869961 seconds (182.80 M allocations: 11.127 GiB, 3.79% gc time, 10.18% compilation time)
julia> @time include("./makie.jl")
Starting Program
Initialize the figure
del_t = 0.009360074996948242
Display the figure
del_t = 0.002168893814086914
Get the figure Axis
del_t = 0.06691288948059082
Plot …
del_t = 0.0012462139129638672
Leaving program
0.109441 seconds (682.27 k allocations: 18.793 MiB, 38.28% compilation time)
julia>
Thus, all the suggestions here are reasonable (adapt your workflow to not restart Julia all the time, use revise, maybe a sysimage). The relationship with vscode may be that the command you are using to run the code from within it is restarting the Julia section every time.
It’s true it doesn’t take very long to make a sysimage for my typical project (I just tried it took 10-15 minutes). But I’m not too worried about the occasional upgrade. The problem is for the various packages I add when I find the need, in the middle of the project. It would be disruptive to make a new sysimage when I need an OrderedDict, then later in the day when I need AxisKeys, etc.
There’s also the problem that Julia encourages a separate environment for each project. I think that’s a very good thing, but it means I need to make a sysimage for each project, to have a self contained, reproducible environment (which of course I want because Julia has made me used to it )
Here’s an example from today: I just got a small dataset. I want to look at it with AlgebraOfGraphics (using CairoMakie). But to do things well I need to make a new sysimage, and I need to install a new IJulia kernel (hand-tuned to use that sysimage), just for this little project
I have had no problems adding packages. For example I add LazyArrays or Infiltrator or whatever to projects and just use my data analysis sysimage. So far it’s not been a problem.
Wow! I appreciate the great comments. Thanks sdanishch for the GtkReactive vs Makie explanation and your VSCode comments. David Anthoff’s video gave me some new ideas until he went into areas irrelevant to me.
I am very pleased with what I’ve accomplished already using Makie from VSC interactively. I’m slowly working through developing up test cases for what I want to do in making my program interactive from a GUI vice just mouse/keyboard interactions. I’ve a lot to learn but having fun.
I think it should be possible to pin all versions used in the sysimage so that you can only add packages compatible with the existing stack. You might not get the newest versions if your sysimage is older but it should keep you from reaching an invalid environment.
Good idea… I tried with pin --all and it only pins the explicit dependencies (those listed in Project.toml). I would have to manually add pinned = true to all the other packages in Manifest.toml and check that this is respected by Pkg.
This should really be the default behavior (or rather, the only behavior ): do not touch the version of packages in the sysimage.
I would not be comfortable doing that for serious code. Here’s an analogy: In general if I see a piece of code that is documented to work 99% of the time (and fail silently the remaining 1%), that’s a bug. It should be replaced with code that always work even if that’s slower.
Yeah but also 10 minutes each time I want to add a small package… Also I’m working simultaneously on multiple “serious” projects, so I have to keep track of many sysimages.
Then there’s the confusion in IJulia: for each sysimage I install a custom kernel configured to use it. It’s a bit of a mess when I want to create a new notebook: IJulia will show all the kernels and I have to pick the correct one. It’s already bad with a single sysimage: in any project folder, IJulia will offer to create a notebook with that kernel even if the project is not compatible with the sysimage.
)