How to keep Julia dependencies precompiled between terminal sessions?

New to Julia
, ,
1

Hello everyone,

I’m developing a Julia-based simulation model that runs from the terminal (no UI). Each time I execute the script, it precompiles all dependencies, which takes quite a long time (around 8 minutes).

Is there a way to keep installed dependencies (e.g., JuMP, Agents) precompiled and cached so that subsequent runs only compile my own modules, rather than recompiling all packages each time? Ideally, I’d like behavior similar to running code from VS Code, where even between sessions it does not recompile everything only the custom modules.

Edit:

I tried the solution from this thread but it did not work for me.

Here are the dependencies I have (Revise only for development)

[deps]
Agents = "46ada45e-f475-11e8-01d0-f70cc89e6671"
CPUTime = "a9c8d775-2e2e-55fc-8582-045d282d599e"
Coluna = "88b4ec78-b192-11e8-04aa-4d367dd96a64"
DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
GLMakie = "e9467ef8-e4e7-5192-8a1a-b1aee30e663a"
Gurobi = "2e9cd046-0924-5485-92f1-d5272153d98b"
HiGHS = "87dc4568-4c63-4d18-b0c0-bb2238e4078b"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
JuMP = "4076af6c-e467-56ae-b986-b466b2749572"
MathOptInterface = "b8f27783-ece8-5eb3-8dc8-9495eed66fee"
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
Timers = "21f18d07-b854-4dab-86f0-c15a3821819a"

Here is the terminal output each time I run the code (the 29 changes from one run to the another without any modifications of the codebase):

Precompiling project...
  Progress [==>                                      ]  1/29
  ◓ Latexify → DataFramesExt
  ◓ TiffImages
  ◑ MathTeXEngine
  ◓ Distributions → DistributionsChainRulesCoreExt
  ◒ BlockDecomposition
  ◐ StructArrays → StructArraysStaticArraysExt
  ◐ Accessors → StructArraysExt

Precompiling project...
  19 dependencies successfully precompiled in 368 seconds. 374 already precompiled.
2

Since I’m also writing my own code. From my side, I observe it appears that only if I had modified some modules would julia re-compile, e.g.

julia> using EM, EM.In
Precompiling EM finished.
  1 dependency successfully precompiled in 4 seconds. 52 already precompiled.
3

Odd, precompile caching is the default behavior and it is usually only overwhelmed by frequent-enough environment switching. Just to check our terminology here, are you seeing a Precompiling printout each session?

4

In my experience, frequent precompilation has often to do with code loading from shared environments before activating the actual environment (for example when you load Revise in your startup and activate the env within your script). See also

1 Like
5

How are the script’s dependencies set up? If they are not in an environment and you create a temp environment then they cannot be cached.
Please post how exactly you run the script and the beginning of the script where you load the libraries you want to use.

6

Same problem here: How to achieve consistency when using workspaces on Julia 1.11? - #2 by ufechner7 And no, I am not using Revise. In my case, it only happens with Julia 1.11, not with Julia 1.12.

1 Like
7

I update the post with more info and as shown it is precompiling everything (I guess) even when I don’t modify anything in the code.

8

Yes I updated the post with more details, as shown each time I run my code it precompiles everything even when I don’t modify anything. I tried a solution from an old thread but it did not work for me.

9

Yes I do use Revise for development and my script activate the environment and check for missing packages because it will be used by other people in the future so I have to make sure they have all the dependencies. I will try to remove Revise and see what happens. Thank you

10

I updated the post with the Project.toml content, the program uses a combination of local modules and installed dependencies and because the program will be used by other people each time I run the code Main.jl it executes this part to check for packages:

using Pkg

Pkg.activate(".")

Pkg.instantiate()

Besides this part nothing fancy happens it executes the main() which is just an encapsulation for some function calls from the project.

Note: I do use Revise.jl for development which I will try to remove and check if it resolves the issue.

11

So after all the solution was just removing Revise and it run as intended quickly and did not precompile everything each run. Thank you for your help.

12

If you load Revise.jl in your startup.jl then simply execute the script with jullia --startup=no script.jl

2 Likes
13

I think the actual solution is to always execute scripts with

julia --project=@. script.jl

(possibly replace @. with whatever path your environment lives at) and basically never Pkg.activate during a session. This will activate the correct environment before startup.jl is executed, therefore statements like using Revise are evaluated/resolved in the correct environment and the state of your main env manifest does not leak into your actual environment.

This is a very common footgun with cached precompilation + stackend environments. Personally, I think activate should warn each time you activate an environment whose manifest is partly ignored because different versions of packages are already loaded (warnings like this sometimes appear during precompilation).

3 Likes