Sharing precompiled code across docker images

racinmat
Wed, 15 Sep 2021 21:07:24 +0000
Performance

I’m struggling with the precompilation time in julia inside docker image.
We have some projects in julia in docker, and we’re doing changes quite often to this project, but the packages that take most time to precompile (Flux etc.) change version only rarely.
I’m thinking whether there is a way to store the precompilation result and when new docker image is built, it would check if it has already been precompiled or not.
I know PackageCompiler states in docs that using sysimage would work only on the same machine it was created on, would it work in same docker image, so I could store the sysimage if it’s not created, and if it’s already stored somewhere, I would load it as part of docker build?
When I produce sysimage using PackageCompiler, and store these sysimages for various versions of julia and versions of packages if there a way to determine if I already have the sysimage ready for my versions of packages and I can load it instead of building it again?
Would conditioning sysimage by version of julia and versions of all packages I precompile there be enough to determine if it’s safe to load such sysimage because it’s been created with same versions of packages and thus should be the same?

In order to make this work, of course the sysimage would need to be built on same/similar OS and cpu (and gpu if there would be any) supporting same instructions, right?

Is PackageCompiler capable of producing multiple sysimage targets, so it would contain code for more architectures and the most performant would be selected at runtime?

As a low-hanging fruit I could just cache whole ~/.julia. But I hope there would be some handier way to work with it, but probably caching the ~/.julia seems like a low-hanging fruit.
And I think ~/.julia is too big, in the end I need only ~/.julia/compiled/<my version of julia> , right?
Each package in e.g. ~/.julia/compiled/v1.5 has several *.ji files, is there any way to check per-file/package what would be the resulting name of .ji file and fetch it if it’s been pre-computed, and if not, then precompile that package?

6 posts - 6 participants