Everything precompiling when opening new notebook

General Usage Performance
1

Yesterday I was in the machine learning workshop and following along having successfully followed all the setup instructions. At some point we were asked to open a new notebook and it took over 11 minutes before I’d done anything. Opening the notebook seemed to start the process of precompiling everything in my 1.7 environment.

I’ve been looking around for answer to this problem but they seem to focus on package development or state that when starting up a new session (if that’s the right word) everything required by your notebook has to be compiled and that plotting packages take a while… but there wasn’t anything in my notebook. Ok, maybe if using Pluto it has to compile that… but everything, for 11 minutes before I could do anything?

Clearly this problem is not happening for other people because it would make Julia completely unusable so I don’t understand what they’ve done differently.

This was most noticeably a pluto issue, but I tried opening a jupyter notebook and it did the same thing (maybe due to the workshop configuration?). Jupyter notebooks and just plain .jl files have been usable for me in the past.

Is it time to nuke everything and reinstall from scratch?

Thanks.

2 Likes
2

Could you try starting a command line REPL and see if the issue can reproduced there after the second session? Also could you show us some of the output. It may be possible that you are repeating some unnecessary steps in the process.

What should happen is that precompilation should occur only once and be cached in the directory ~/.julia/compiled/v1.7/. It seems that on your system this caching is not occurring for some reason.

1 Like
3

Starting a new julia repl and launching pluto with a new notebook was ok so I’m going through the workshop setup again to see if I can reproduce the problem that way.

It has a setup() function that has activated a dev environment, done some precompilation and is now running PackageCompiler…

I think this is what made the notebooks that were part of the workshop faster to load.

Will try your suggestion and get back… though I suspect that you are right that I switched environment when I attempted to use a jupyter notebook instead of the pluto one and it went through the same process again. Whether simply opening a new notebook from this environment will trigger the problem we shall see…

4

Running PackageCompiler will take a while, but that is something you probably only need to do once if at all depending on the usage pattern. After the initial use of PackageCompiler, you should be able to pass the compiled system image to Julia via the -J or --sysimage option. Using PackageCompiler is not something that you should be doing in every notebook session. Frankly, this is something I usually only do when moving code to production.

Note that the use of PackageCompiler is distinct from what we usually refer to as “precompilation”. PackageCompiler.jl involves ahead-of-time compilation of Julia to native code. This is stored in a native shared library with a .dll, .so, or .dylib extension. Precompilation refers to parsing and type inference steps done before native code compilation. Precompilation is cached in the ~/.julia/compiled folder in .ji files.

1 Like
5

Running PackageCompiler isn’t something I’d normally do, but today I have not reproduced the problem, the first run of setup() produced output like this

pluto()
  Downloaded artifact: MKL
Starting a Pluto notebook in your browswer.
Please use `HelloJulia.stop()` to interrupt the Pluto notebook server.

julia>       From worker 2:	[ Info: Loading...
      From worker 2:	    Updating registry at `~/.julia/registries/General.toml`
      From worker 2:	┌ Info: 
      From worker 2:	└ Opening http://localhost:1234/open?secret=iuJ0GDAs&path=%2Fhome%2Fandy%2F.julia%2Fdev%2FHelloJulia%2Fnotebooks%2Fpluto_index.jl in your default browser... ~ have fun!
      From worker 2:	┌ Info: 
      From worker 2:	│ Press Ctrl+C in this terminal to stop Pluto
      From worker 2:	└ 
      From worker 2:	Gtk-Message: 10:15:24.942: Failed to load module "canberra-gtk-module"
      From worker 2:	Gtk-Message: 10:15:24.948: Failed to load module "canberra-gtk-module"
      From worker 2:	    Updating registry at `~/.julia/registries/General.toml`
      From worker 2:	   Resolving package versions...
      From worker 2:	   Installed KernelDensity ─ v0.6.4
      From worker 2:	    Updating `~/tmp/jl_hK1hhk/Project.toml`
      From worker 2:	  [13f3f980] + CairoMakie v0.8.10
      From worker 2:	    Updating `~/tmp/jl_hK1hhk/Manifest.toml`
      From worker 2:	  [621f4979] + AbstractFFTs v1.2.1
      From worker 2:	  [1520ce14] + AbstractTrees v0.4.2
      From worker 2:	  [79e6a3ab] + Adapt v3.3.3
      ...

Then opening a new repl and running pluto (it was fine so tried using CairoMakie)

julia> Pluto.run()
[ Info: Loading...
┌ Info: 
└ Opening http://localhost:1235/?secret=0mfAY02w in your default browser... ~ have fun!
┌ Info: 
│ Press Ctrl+C in this terminal to stop Pluto
└ 
Gtk-Message: 10:20:47.712: Failed to load module "canberra-gtk-module"
Gtk-Message: 10:20:47.714: Failed to load module "canberra-gtk-module"
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/tmp/jl_DJ5aU2/Project.toml`
  [13f3f980] + CairoMakie v0.8.10
    Updating `~/tmp/jl_DJ5aU2/Manifest.toml`
  [621f4979] + AbstractFFTs v1.2.1
  [1520ce14] + AbstractTrees v0.4.2
  [79e6a3ab] + Adapt v3.3.3
  [27a7e980] + Animations v0.4.1
  [67c07d97] + Automa v0.8.2
  ...

This time all in green with + instead of compiling again, which is what I would have expected.

Thanks for your help, I think this will require some more investigation on my part. I do recall when shutting things down last night that, even after all julia windows were closed, a julia process was stuck running 100% cpu that I had to kill. Maybe something just got screwed up yesterday and I was so used to slow first-time-to-plot issues that I thought it was all part of the same problem.

Until the workshop yesterday I’d abandoned Pluto as it regularly took over 5 minutes to load a small notebook. I’m going to re-read 5. Creating Packages · Pkg.jl as I think I might have a better understanding now of what’s happening.

I feel like a lot of the getting started tutorials for julia skip over the intricacies of package management, which is understandable, but then people new to julia get bitten, and you can see that the person giving the demonstration is waiting just a few seconds for their notebook to load while yours is taking minutes - anyway, that’s just me griping now.

Is it at all advisable to have a lean general environment (e.g. v1.7) and use specific package environments for each task, even if just trying things out and learning Julia?

If I’ve understood correctly, an update to one package will cause a recompilation to all its dependencies, so maybe there is some good practice of running updates and precompiling your environment again; when you’re off to get a coffee and not when you’re in the middle of a task?

There is a performance tips page in the manual, though that didn’t seem relevant to this kind of problem, is there any general guide for workflows in Julia or is it simple enough to say that when you’ve launched a repl you should keep it open - all my programs and notebooks are a lot faster after the first run?

6

Yes. I usually have almost nothing in my base environment (e.g v1.7). If I do have something there it might be Revise and BenchmarkTools, utilities that I might want in every environment. For messing around I usually use a temporary environment.

using Pkg
pkg"activate --temp"
1 Like
7

It could, yes. In development, Revise.jl can mitigate this effect. It’s best to keep environments minimalist. Also note that environments stack:

https://docs.julialang.org/en/v1/manual/code-loading/#Environments-1

I usually use the Revise-based workflow described below.

https://docs.julialang.org/en/v1/manual/workflow-tips/#Revise-based-workflows

@Erik_Engheim nicely illustrates this here:

8

Funnily enough, while following your advice to use a temp environment I found another thread which discusses many of the same issues.

Whatever went wrong specifically yesterday in the workshop, it does seem that I’m generally not using Julia the best way.

Thanks again for all your help, I shall try out these suggestions, this definitely feels like the “secret” I was missing when I couldn’t understand why I was running into these issues. In hindsight I probably should have posted to the newbies thread.

9

Perhaps this was added later, but this would quickly remove all packages in the environment.

using Pkg
pkg"rm --all"

  [rm|remove] [-p|--project] pkg[=uuid] ...
  [rm|remove] [-p|--project] [--all]

  Remove package pkg from the project file. Since the name pkg can only refer
  to one package in a project this is unambiguous, but you can specify a uuid
  anyway, and the command is ignored, with a warning, if package name and UUID
  do not match. When a package is removed from the project file, it may still
  remain in the manifest if it is required by some other package in the
  project. Project mode operation is the default, so passing -p or --project
  is optional unless it is preceded by the -m or --manifest options at some
  earlier point. All packages can be removed by passing --all.

  [rm|remove] [-m|--manifest] pkg[=uuid] ...
  [rm|remove] [-m|--manifest] [--all]

  Remove package pkg from the manifest file. If the name pkg refers to
  multiple packages in the manifest, uuid disambiguates it. Removing a package
  from the manifest forces the removal of all packages that depend on it, as
  well as any no-longer-necessary manifest packages due to project package
  removals. All packages can be removed by passing --all.

Also perhaps you like the current environment and would simply not have it be the default environment. You might be interested in