Faster setup for a new machine?

At my work, we develop and run code on AWS ec2 machines, and often spin new ones with the same environment. We have both a big Python and Julia environment, but the Julia portion of the setup Pkg.instantiate() is by far the slowest step, taking 30+ minutes to download all the artifacts and precompile all our (many) dependencies.

Given that I have pretty much the exact same environment in several machines, is there a way to speed this up? E.g. by transferring the precompiled files and artifacts to some location before running instantiate?

Have you considered PackageCompiler.jl ?

I’ve used it to create sysimages before, but my understanding is that the environments in those are frozen. And when I spin up a new machine, I want to be able to add and update packages, just using the common environment as a baseline.

1 Like

Yeah, PackageCompiler.jl is not flexible in that way. I have seen chached file relocation but did not try it. Hopefully this helps or others have better options.

[untested] just zip/unzip an instantiated .julia folder??

7 Likes