I think the long-term approach would be to use hash tree (Merkle tree) to generate the path of precompilation cache file. That is to say, the hash value of a given package depends on:
-
“Content” of the package which may be represented by one of the following:
- (Option A) its UUID, version, and package options
- (Option B) paths of all source files and files specified by
include_dependency
-
Hash value of all dependencies.
Given the hash value, the path to the compilation cache would be ~/.julia/compiled/v$X.$Y/$package_name/$digest_of_the_hash.ji as done today.
I think the hardest part is how to retrieve the hash value of all dependencies. This is difficult because you have to know the dependencies before start loading the package (kind of a chicken-and-egg problem). It’s difficult especially because dependency tree in Julia is dynamic; i.e., you can (de)activate projects in the middle of session. IIUC, this may choose different set of versions of packages depending on the order of imports and (de)activations. So, I think you need to build in-memory dependency tree and propagate it to subprocesses that precompile the packages.
Alternatively, maybe you can remove Pkg.activate (or highly restrict its usage) so that the whole dependency tree is statically determined by Manifest.toml (provided that it’s not modified by other processes). With this approach you need to add Manifest.toml loader in Base (which probably requires to add Base.Toml module). Another way to retrieve dependencies is to cache the dependencies recorded during precompilation in a persistent database file for each project.