I am trying to write an install script that instantiates and precompiles the main project and all sub-projects. It seems to work with Julia 1.12, but not with Julia 1.11. Is there a way to make it work with Julia 1.11?
I tried:
First to copy Manifest-v1.11.toml.default to Manifest-v1.11.toml (the de9.fault manifest is from Git and represents the last known good set of package versions
Instantiate the project
julia --project -e '
using Pkg
try
Pkg.instantiate()
catch e
@warn "Pkg.instantiate() failed, attempting fresh resolve..." exception=e
proj_dir = dirname(Base.active_project())
# Remove all manifest files to force a clean resolve
for f in readdir(proj_dir)
if startswith(f, "Manifest") && endswith(f, ".toml") && !endswith(f, ".default")
rm(joinpath(proj_dir, f), force=true)
end
end
Pkg.resolve()
Pkg.instantiate()
end
'
Precompile the main project:
julia --project -e 'using Pkg; Pkg.precompile()'
Resolve sub-projects independently:
rm -f examples/Manifest*.toml test/Manifest*.toml examples_3d/Manifest*.toml
julia --project=examples -e 'using Pkg; Pkg.resolve(); Pkg.instantiate()'
echo "Precompiling examples project..."
julia --project=examples -e 'using Pkg; Pkg.precompile()'
julia --project=test -e 'using Pkg; Pkg.resolve(); Pkg.instantiate()'
echo "Precompiling test project..."
julia --project=test -e 'using Pkg; Pkg.precompile()'
julia --project=examples_3d -e 'using Pkg; Pkg.resolve(); Pkg.instantiate()'
echo "Precompiling examples_3d project..."
julia --project=examples_3d -e 'using Pkg; Pkg.precompile()'
While this somewhat works, it results in:
the sub-projects not using the same package versions as the main project
If I open any of the examples, it starts precompiling the packages again
Any idea how to fix these issues?
Observation:
If I copy the main manifest into a subproject, activate the subproject, and run Pkg.resolve(), this often fails. Is this a bug in the Pkg package?
I can’t imagine this working without workspaces, which were introduced in 1.12. I haven’t heard of any efforts to backport it, which is rare for features generally, but it’s very odd the Pkg docs don’t mention anything to the effect of “1.12 onwards”.
This is a fundamental problem which can’t be solved by copying manifestes. Consider the following:
root dir env only depends on PkgA with compat=[1,2]. If you resolve this env, you will get PkgA at version 2.
sub env depends on PkgA and PkgB. PkgB is only compatible with PkgA@v1. Therefore, you sub env will get PkgA@v1.
Both of those are resolved correct, but they are just incompatible.
The “workspaces” feature (which certainly won’t be backported) essentially creates a virtual environment which merges all dependencies and compats of all sub envs. Therefore, you will get PkgA@v1 even in the root dir.
If you want similar behavior, you could create an additional dev env for 1.11:
where you essential autogenerate a Project.toml by parsing the sub envs and combining all dependencies. Then you’d need to make sure that you always start your julia 1.11 sessions like julia --project=/path/to/dev_1.11_env. This dev env would be a true superset of all subenv envs and therefore could run all code in the subenvs while caching the precompilation.
Caveats:
the merging of the differen Project files might be non-trivial, especially if compat bounds are involved
you won’t be able to “add” packages to the subenvs, because you never activate them
The other obvious alternative is to resolve all envs separatly and precompile all envs separatly once during ./install. This takes longer, but no matter what subenv julia --project=../subenv activates, there will be a valid cache for that env already.
delete and re-add the main registry (yes, it is sometimes needed, students have computers where it is broken)
copy Manifest-v1.11.toml.default to Manifest-v1.11.toml
resolve and instantiate the main environment in a try-catch block; if it fails, delete the manifest and resolve and instantiate again. In that case, rename Manifest.toml to Manifest-v1.11.toml
copy Manifest-v1.11 to all sub-environments, but not to docs environment
do the same in each sub environment
So far, this seems to work on Linux and Windows, tests on MacOS still ongoing.
I can now run examples from each sub-environment and nothing gets pre-compiled, all the compilation happens in the install script.
I understand that this might not work well if there are conflicting requirements for the sub-projects. But in my project, this should not be the case.
I had to manually downgrade SciMLBase from 2.144.2 to 2.144.0 in Manifest-v1.11.toml.default, though.