How to reuse julia compiled cache across machines?

schlichtanders · July 17, 2024, 3:19pm

Hello,

I’d like to cache the julia package compilation in an external volume which can be read and written to by multiple clients (using a respective kubernetes volume) .

My question is whether it is enough to bring the .julia/compile folder to this special volume, or whether something else, e.g. if the lockfile resides somewhere else, needs to be brought there too.

Thanks to the following two julia additions I hope this works in principle since 1.10

precompile: respect and report cache pidlocking by IanButterworth · Pull Request #3519 · JuliaLang/Pkg.jl · GitHub
pidlock cache file precompilation by IanButterworth · Pull Request #49052 · JuliaLang/julia · GitHub

ianshmean · July 17, 2024, 4:20pm

It should just work as you describe.

vchuravy · July 17, 2024, 5:52pm

Modulo different CPU architectures across the clients. You might want to set JULIA_CPU_TARGET appropriately

schlichtanders · July 17, 2024, 8:55pm

thanks for the hint

schlichtanders · August 13, 2024, 4:22pm

I tried this now but experience recompilation despite the compiled directory being reconstructed. I guess another directory is needed somehow for caching compilation to work…

Does someone know which other package is needed? Or how to debug?

schlichtanders · August 13, 2024, 4:59pm

It seems that it is due to mtime changes inside .julia/packages looking into the code it seems that in principle julia has different ways of caching

github.com

JuliaLang/julia/blob/b7aa5e37402e2a0c81f9f2b80e89cf0f85ff6da6/base/loading.jl#L3828-L3855


      
          if ftime_req >= 0.0
              # this is an include_dependency for which we only recorded the mtime
              ftime = mtime(f)
              is_stale = ( ftime != ftime_req ) &&
                         ( ftime != floor(ftime_req) ) &&           # Issue #13606, PR #13613: compensate for Docker images rounding mtimes
                         ( ftime != ceil(ftime_req) ) &&            # PR: #47433 Compensate for CirceCI's truncating of timestamps in its caching
                         ( ftime != trunc(ftime_req, digits=6) ) && # Issue #20837, PR #20840: compensate for GlusterFS truncating mtimes to microseconds
                         ( ftime != 1.0 )  &&                       # PR #43090: provide compatibility with Nix mtime.
                         !( 0 < (ftime_req - ftime) < 1e-6 )        # PR #45552: Compensate for Windows tar giving mtimes that may be incorrect by up to one microsecond
              if is_stale
                  @debug "Rejecting stale cache file $cachefile because mtime of include_dependency $f has changed (mtime $ftime, before $ftime_req)"
                  record_reason(reasons, "include_dependency mtime change")
                  return true
              end
          else
              fstat = stat(f)
              fsize = filesize(fstat)
              if fsize != fsize_req
                  @debug "Rejecting stale cache file $cachefile because file size of $f has changed (file size $fsize, before $fsize_req)"
                  record_reason(reasons, "include_dependency fsize change")

This file has been truncated. show original

the mtime way is only one, the other being hash based, which would be better here for central caching.

Does someone know what needs to be set/available for precompilation to use hash for identification?

EDIT: I think I understood that just some packages specify explicitly to use mtime instead of hash by specifying include_dependency("myexternal.file", track_content=false). Hence it is indeed necessary to also backup the packages subdirectory for caching to work.

ufechner7 · August 13, 2024, 5:55pm

This should work better in Julia 1.11 (not yet released), for example because of Avoid repeated precompilation when loading from non-relocatable cachefiles by fatteneder · Pull Request #53905 · JuliaLang/julia · GitHub .

fatteneder · August 13, 2024, 8:12pm

To load the caches you need compiled and packages.

Hash identification only works with 1.11 onwards, and that is needed to allow moving the compiled folder.

To debug re-precompilation issues run Julia with the environment variable JULIA_DEBUG=loading.

include_dependency("myexternal.file", track_content=true) is now the default.

Here are some notes on how to relocate pkgs on 1.11: PSA: Cache relocation with Julia v1.11 - HackMD

schlichtanders · August 14, 2024, 10:54am

I know tried to symlink the compiled folder, which works, and also the packages folder to a cache directory, which unfortunately kind of does not work.
It makes julia being super super super slow, like almost not responsive. Not sure what is going on…

EDIT: I think it is not because of the symlink but because of some characteristics of my remote filesystem which is not fully posix compliant or maybe slow for some things.