At my work, we develop and run code on AWS ec2 machines, and often spin new ones with the same environment. We have both a big Python and Julia environment, but the Julia portion of the setup Pkg.instantiate() is by far the slowest step, taking 30+ minutes to download all the artifacts and precompile all our (many) dependencies.
Given that I have pretty much the exact same environment in several machines, is there a way to speed this up? E.g. by transferring the precompiled files and artifacts to some location before running instantiate?
I’ve used it to create sysimages before, but my understanding is that the environments in those are frozen. And when I spin up a new machine, I want to be able to add and update packages, just using the common environment as a baseline.
Yeah, PackageCompiler.jl is not flexible in that way. I have seen chached file relocation but did not try it. Hopefully this helps or others have better options.