Recompiling stale cache files


#1

Hello dear Julia users!

I really like this language due to its speed and usability. However, a lot of times my modules are recompiling, which can take a long time when my script is starting. I read here and there some stuff about the conditions, when a cache is forced to recompile. But I could not remember every condition, nor did I find the rational behind it, why the cache has to recompile under these conditions. (That the cache has to recompile when I use a different Julia version or changed a file of the module is clear. Other conditions looked a bit mysterious to me). Could you provide me with a complete list with the conditions, when a cache has to recompile? And the rational behind it, so I will understand it and remember?

Furthermore, I would really appreciate some tips and tricks how to make my modules (or modules added with the package manager) not recompile every now and then. I am using Atom with Juno for development, and sometimes I run the full skript with the Play button, sometimes I do inline evaluation, and sometimes I type something in the REPL in Atom. Sometimes I also exit() and restart Julia in the Atom REPL to get rid of the current workspace. To me it seems, the modules recompile at random occasions. I would really like to understand this behavior and find a way to work around this. Because waiting so long every time starts to get annoying.


#2

Recompilation should only need to happen after you have updated packages or changed your julia version. Does it happen more than that?

For packages being developed, it sometimes makes sense to put __precompile__(false) on top of them to make them not precompile, since we want to reload them over and over.


#3

Yes, it seems it does recompile more often, but not every time. I will keep a closer look on it, then I can report back. However I could not find any reproducible step yet.
I also will look for the precompile flag. Does it default to true, when it is not set?


#4

Yes.


#5

Right now I had the case, that the library ControlSystems did recompile. Though I did not update anything since the last time I used it (some seconds ago). But I don’t know how to reproduce this. It occurs very unpredictable, but luckily also very seldom.

However, what suprprises me as well is that the first loading of a Module takes quite long. Why does it take so long and what is it doing in this time? Does it take so long to load all the functions in the RAM? And why does it take some time at the second run as well, and only the third run is very fast?

@time using ControlSystems

8.920345 seconds (17.39 M allocations: 957.472 MiB, 5.97% gc time) [first run]
1.846112 seconds (4.10 M allocations: 196.812 MiB, 4.04% gc time) [second run]
0.000295 seconds (293 allocations: 15.281 KiB) [third run]

#6

Can I actually see the reason, why a cache is recompiling, somewhere? Maybe in a Pkg or Julia debug version?
And since I forgot to mention it: I am using Julia 1.0.3


#7

UPDATE:

I have a reproducible scenario now:
I have installed the module QuadGK (in the official registry) in both the default environment (v1.0) as well as in Environment (EnvA). Executing the commands in the following sequence, the package will compile EVERY TIME, although none of the Julia or Package versions did change:

$ julia
> using QuadGK
[ Info: Recompiling stale cache file /home/maximilian/.julia/compiled/v1.0/QuadGK/hq5Ol.ji for QuadGK [1fd47b50-473d-5c70-9696-f719f8f3bcdc]
> exit()
$ julia
] activate EnvA
> using QuadGK
[ Info: Recompiling stale cache file /home/maximilian/.julia/compiled/v1.0/QuadGK/hq5Ol.ji for QuadGK [1fd47b50-473d-5c70-9696-f719f8f3bcdc]
> exit()
$ julia
> using QuadGK
[ Info: Recompiling stale cache file /home/maximilian/.julia/compiled/v1.0/QuadGK/hq5Ol.ji for QuadGK [1fd47b50-473d-5c70-9696-f719f8f3bcdc]
# and so on and so forth......

Notice, that the Version and the Path to the QuadGK module did not change. This only happens, when alternating between the default environment and a custom environment. When creating two custom environment (EnvA and EnvB for example), and alternating between these ones, the package will NOT recompile. Also, I tried to do this with the package Example, but this package did also not reproduce this behaviour. But QuadGK is not the only package, it happens with quite some packages, and one of them needs to compile very long.

So, where is the Problem: Is it still me? Is it the Package Manager (I am using Julia 1.0.3), or is it the module QuadGK? Should I write a bug ticket, and if so, where? (On slack I got pointed to this issue: https://github.com/JuliaLang/julia/issues/27418 . Is this related to the behaviour I am experiencing? So is this problem already been known and worked on?)

By the way: When I installed the package Example in my default environment, it updated all the packages I have installed there. I thought, the new package manager wouldn’t do this any more. Why did I experience this?

(v1.0) pkg> add Example
 Updating registry at `~/.julia/registries/General`
 Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Installed Mustache ───────── v0.5.10
Installed Homebrew ───────── v0.7.1
Installed ChaosTools ─────── v1.3.1
Installed DelayEmbeddings ── v0.3.0
Installed HDF5 ───────────── v0.10.4
Installed NearestNeighbors ─ v0.4.3
 Updating `~/.julia/environments/v1.0/Project.toml`
 [7876af07] + Example v0.5.1
 Updating `~/.julia/environments/v1.0/Manifest.toml`
 [608a59af] ↑ ChaosTools v1.3.0 β‡’ v1.3.1
 [864edb3b] ↑ DataStructures v0.14.0 β‡’ v0.15.0
 [5732040d] ↑ DelayEmbeddings v0.2.0 β‡’ v0.3.0
 [eb300fae] ↑ DiffEqBiological v3.4.1 β‡’ v3.4.2
 [459566f4] ↑ DiffEqCallbacks v2.4.0 β‡’ v2.5.0
 [7876af07] + Example v0.5.1
 [28b8d3ca] ↑ GR v0.36.0 β‡’ v0.37.0
 [f67ccb44] ↑ HDF5 v0.10.3 β‡’ v0.10.4
 [d9be37ee] ↑ Homebrew v0.7.0 β‡’ v0.7.1
 [ffc61752] ↑ Mustache v0.5.8 β‡’ v0.5.10
 [b8a86587] ↑ NearestNeighbors v0.4.2 β‡’ v0.4.3
 [1dea7af3] ↑ OrdinaryDiffEq v4.18.3 β‡’ v4.19.0
 [69de0a69] ↑ Parsers v0.2.15 β‡’ v0.2.16
 [e409e4f3] ↑ PoissonRandom v0.3.0 β‡’ v0.3.1
 [789caeaf] ↑ StochasticDiffEq v5.10.4 β‡’ v5.11.0
 [c3572dad] ↑ Sundials v2.6.0 β‡’ v2.7.0
 [bd369af6] ↑ Tables v0.1.12 β‡’ v0.1.14
 Building Homebrew β†’ `~/.julia/packages/Homebrew/s09IX/deps/build.log`
 Building HDF5 ────→ `~/.julia/packages/HDF5/R6yOo/deps/build.log`

#8

I’m also getting this with some of my own packages, it seems to happen more when there was some issue when installing the package (I was missing some dependencies in REQUIRE). Is there any solution to this beside deleting everything and starting fresh ? Can I save my environment and just instantiate maybe ?


#9

The package also needs to recompile if any of the dependencies changes version. Maybe those are different between the custom and the default environment?


#10

Hm, this is a good hint! I don’t know. I need to check this. Thanks for the input!