Understanding why I am seeing precompilation on github actions. Cache partial miss

General Usage
1

I am seeing this message in my github actions logs:

  julia scripts/julia_dev_tooling/format.jl
  shell: /usr/bin/bash -e {0}
  env:
    JULIA_PKG_SERVER_REGISTRY_PREFERENCE: eager
  Activating project at `~/work/biohaus/biohaus/scripts/julia_dev_tooling`
Precompiling project...
    863.3 ms  ✓ Glob
   1097.5 ms  ✓ Preferences
    510.4 ms  ✓ PrecompileTools
  25113.2 ms  ✓ CommonMark
    918.3 ms  ✓ CommonMark → CommonMarkMarkdownExt
  32835.5 ms  ✓ JuliaSyntax
  46553.2 ms  ✓ JuliaFormatter
  7 dependencies successfully precompiled in 79 seconds. 10 already precompiled.
[ Info: <messages from my script>

full logs

So I am seeing

7 dependencies successfully precompiled in 79 seconds. 10 already precompiled.

So it is at least partially hitting my cache since 10 were already precompiled.
Why is it not fully hitting it? Why are 7 being precompiled?
I can’t see any messages further up about new versions being installed,
and I shouldn’t since i have a manifest checked in, and i haven’t touched my dependencies at all in this PR.

I am using the standard julia-actions/cache action.

2

Setting the environment variable

JULIA_DEBUG=loading

will tell you why existing cache can’t be hit.

3

Thanks.
I had a feeling i had used something like this before.

Now to wait til I see it reoccur

4

Ok.
I am seeing a few causes.
But by far the majority are

Debug: Rejecting cache file /home/runner/.julia/compiled/v1.11/CommonSolve/zEcGf_aN5US.ji for Base.PkgId(nothing, ""):
  Reasons = "Unable to find compatible target in cached code image.\nTarget 0 (znver3): Rejecting this target due to use of runtime-disabled features\n"
  Image Targets =
   1-element Vector{Base.ImageTarget}:
    znver3; flags=0; features_en=(sse3, pclmul, ssse3, fma, cx16, sse4.1, sse4.2, movbe, popcnt, aes, xsave, avx, f16c, fsgsbase, bmi, avx2, bmi2, adx, clflushopt, clwb, sha, shstk, vaes, vpclmulqdq, rdpid, sahf, lzcnt, sse4a, prfchw, xsavec, xsaves, clzero)
  Current Targets =
   1-element Vector{Base.ImageTarget}:
    sapphirerapids; flags=0; features_en=(sse3, pclmul, ssse3, fma, cx16, sse4.1, sse4.2, movbe, popcnt, aes, xsave, avx, f16c, fsgsbase, bmi, avx2, bmi2, avx512f, avx512dq, adx, avx512ifma, clflushopt, clwb, avx512cd, sha, avx512bw, avx512vl, avx512vbmi, waitpkg, avx512vbmi2, shstk, gfni, vaes, vpclmulqdq, avx512vnni, avx512bitalg, avx512vpopcntdq, rdpid, cldemote, movdiri, movdir64b, serialize, tsxldtrk, amx-bf16, avx512fp16, amx-tile, amx-int8, sahf, lzcnt, prfchw, xsavec, xsaves, avxvnni, avx512bf16)
@ Base loading.jl:3818

so this says the we are running on a different CPU architecture to what was running for the one that is in the cache.
We are on a sapphirerapids which is a Intel Xeon 4
the cache was saved on a znver3 which is AMD Zen 3

Probably what is going on is that GitHub Actions is running on whatever machines are available from a big pool on Azure.
And it doesn’t care about the microarchitecture.

The znver3 has a few features that the saphirerapids does not.
clzero, sse4a
(the saphirerapids has a ton of features znver3 doesn’t have)

I believe if we always run julia as julia -C x86-64-v3 (or long julia --cpu-target x86-64-v3) that should tell it to only use features that are present in both of these types of machines.

Not sure how to do that in general however.
Because i can’t pass arguments like --cpu-target to julia-buildpkg, or julia-runtests.
I can pass it for things where i trigger julia manually in my workflow.
Which does include the linting, but does not include the tests.

5

Not sure how to do that in general however.

Answer from @vchuravy :slight_smile:
set the JULIA_CPU_TARGET environent variable

6

What did you set it to in order to cover most gh action runners?

7

right now to x86-64-v3
I will give it a few weeks and see if that both

  1. prevents the compilation cache misses and
  2. doesn’t break things.

If it does break things, I will drop down to x86-64-v2, then down to x86-64 then generic.
I suspect x86-64-v3 will be good enough though.

If this does prove to work i will start making noise telling everyone to do this.
And start thinking about how we can make sure it always is set (or otherwise achieved) for all julia use on github actions.

8

What’s nice about JULIA_CPU_TARGET is that you can have multiple targets at the same time, you don’t have to pick the lowest common denominator.

My recommendation is to

  1. take inspiration from the value used in the official builds of Julia
  1. get the values of Sys.CPU_NAME for the machines you want to target (that’s also printed in the output of InteractiveUtils.versioninfo()), and use them.

As a concrete example, if the targets for the x86-64 GitHub-hosted Actions runners are znver3 and saphirerapids you can do

JULIA_CPU_TARGET='generic;sandybridge,-xsaveopt,clone_all;znver3,-rdrnd,base(1);saphirerapids,-rdrnd,base(1)'

generic is the very minimum common denominator, and ensures the cache will be always reusable no matter what specific machines you get for a given architecture, but then you also have specialised native code for all the desired microarchitectures, not just one.

Just be careful that JULIA_CPU_TARGETS has to be set differently for different architectures (for example x86-64 vs x86 vs aarch64), in GitHub Actions you may want to dispatch in on the runner.arch value (but if you use a 32-bit of Julia on a 64-bit machine that won’t be enough).

9

On 1.13+ you can also set JULIA_CPU_TARGET=sysimage to retain the coverage that the sysimage has

10

Based on docker-stacks/images/minimal-notebook/setup-scripts/setup-julia-packages.bash at main · jupyter/docker-stacks · GitHub, FYI

    # https://github.com/jupyter/docker-stacks/blob/main/images/minimal-notebook/setup-scripts/setup-julia-packages.bash
    - name: Set Julia CPU target (x64)
      if: runner.arch == 'X64'
      run: echo 'JULIA_CPU_TARGET=generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1);x86-64-v4,-rdrnd,base(1)' >> $GITHUB_ENV
    - name: Set Julia CPU target (ARM64)
      if: runner.arch == 'ARM64'
      run: echo 'JULIA_CPU_TARGET=generic;cortex-a57;thunderx2t99;carmel,clone_all;apple-m1,base(3);neoverse-512tvb,base(3)' >> $GITHUB_ENV