CI Testing hangs on MacOS nightly

How can I get any diagnostic information on why this one CI test seems to be hanging while all the others pass as expected?

in progress checks
CI / Julia nightly - macOS-latest - default - pull_request (pull_request)Started 39 minutes ago β€” This check has started...

successful checks
CI / Documentation (pull_request)Successful in 1m
CI / Julia 1 - macOS-latest - default - pull_request (pull_request)Successful in 3m
CI / Julia 1 - ubuntu-latest - default - pull_request (pull_request)Successful in 5m
CI / Julia 1 - windows-latest - default - pull_request (pull_request)Successful in 6m
CI / Julia 1.8 - macOS-latest - default - pull_request (pull_request)Successful in 3m
CI / Julia 1.8 - ubuntu-latest - default - pull_request (pull_request)Successful in 4m
CI / Julia 1.8 - windows-latest - default - pull_request (pull_request)Successful in 4m
CI / Julia 1.9 - macOS-latest - default - pull_request (pull_request)Successful in 3m
CI / Julia 1.9 - ubuntu-latest - default - pull_request (pull_request)Successful in 3m
CI / Julia 1.9 - windows-latest - default - pull_request (pull_request)Successful in 4m
CI / Julia lts - macOS-latest - default - pull_request (pull_request)Successful in 3m
CI / Julia lts - ubuntu-latest - default - pull_request (pull_request)Successful in 3m
CI / Julia lts - windows-latest - default - pull_request (pull_request)Successful in 4m
CI / Julia nightly - ubuntu-latest - default - pull_request (pull_request)Successful in 6m
CI / Julia nightly - windows-latest - default - pull_request (pull_request)Successful in 6m
CI / Julia pre - macOS-latest - default - pull_request (pull_request)Successful in 4m
CI / Julia pre - ubuntu-latest - default - pull_request (pull_request)Successful in 5m
CI / Julia pre - windows-latest - default - pull_request (pull_request)Successful in 6m

codecov/patch β€” 95.89% of diff hit (target 93.46%)
codecov/project β€” 94.37% (+0.91%) compared to 168f7af

Clicking on the 3 dots to the right hand side and asking for more information gives

Run julia-actions/julia-buildpkg@v1                                        40m 3s
Run echo "JULIA_PKG_SERVER_REGISTRY_PREFERENCE=${JULIA_PKG_SERVER_REGISTRY_PREFERENCE:-eager}" >> ${GITHUB_ENV}
Run if "false" == "false" && !isdir(DEPOT_PATH[1])

Also, how can I kill this - or will it time-out in the end?

(Edit: Just passed 1hr and still going…)

1 Like

I went out and it was still running 3 hours later when I came back.

I’ve manged to restart by adding a single blank line in the code of the PR, but the behaviour so far is the same.

The task that is hanging (julia-actions/julia-buildpkg@v1) ran in 3s on Julia Pre - MacOS latest and in 6s on Julia 1 - MacOS latest. It seems to be the task the builds the test environment, before even trying to run the package’s test suite.

I don’t have a Mac myself, so can’t explore this locally.

I don’t know how to kill the process, so it’s still running.

We’ve seen this issue in other repos (I also mentioned it on Slack the other day), but I didn’t have the time to investigate

Yes, you can: GitHub - mxschmitt/action-tmate: Debug your GitHub Actions via SSH by using tmate to get access to the runner system itself.

I will look at this but on first inspection I don’t really understand it. Let’s see how I get on!

The test itself was cancelled when it timed out after 6hours last night. The message was as follows:

Run julia-actions/julia-buildpkg@v1
Run echo "JULIA_PKG_SERVER_REGISTRY_PREFERENCE=${JULIA_PKG_SERVER_REGISTRY_PREFERENCE:-eager}" >> ${GITHUB_ENV}
Run if "false" == "false" && !isdir(DEPOT_PATH[1])

[1145] signal 2: Interrupt: 2
in expression starting at none:0
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0) at (unknown file)
__psynch_mutexwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0) at (unknown file)
Allocations: 1 (Pool: 1; Big: 0); GC: 0
Error: The operation was canceled.
For comparison, here is the output of the successful runs on MacOS (e.g. Julia 1)...
Run julia-actions/julia-buildpkg@v1
Run echo "JULIA_PKG_SERVER_REGISTRY_PREFERENCE=${JULIA_PKG_SERVER_REGISTRY_PREFERENCE:-eager}" >> ${GITHUB_ENV}
Run if "false" == "false" && !isdir(DEPOT_PATH[1])
  Installing known registries into `~/.julia`
       Added `General` registry to ~/.julia/registries
    Updating registry at `~/.julia/registries/General.toml`
   Installed InputBuffers ──────────────── v1.1.1
   Installed DataValueInterfaces ───────── v1.0.0
   Installed Tables ────────────────────── v1.12.1
   Installed XML ───────────────────────── v0.3.8
   Installed OrderedCollections ────────── v1.8.1
   Installed TranscodingStreams ────────── v0.11.3
   Installed ColorTypes ────────────────── v0.12.1
   Installed Colors ────────────────────── v0.13.1
   Installed ZipArchives ───────────────── v2.6.0
   Installed DataAPI ───────────────────── v1.16.0
   Installed ArgCheck ──────────────────── v2.5.0
   Installed FixedPointNumbers ─────────── v0.8.5
   Installed CodecZlib ─────────────────── v0.7.8
   Installed Reexport ──────────────────── v1.2.2
   Installed CodecInflate64 ────────────── v0.1.3
   Installed Preferences ───────────────── v1.5.0
   Installed TableTraits ───────────────── v1.0.1
   Installed Statistics ────────────────── v1.11.1
   Installed IteratorInterfaceExtensions ─ v1.0.0
   Installed PrecompileTools ───────────── v1.3.3
    Updating `~/work/XLSX.jl/XLSX.jl/Project.toml`
  [5ae59095] + Colors v0.13.1
  [aea7be01] + PrecompileTools v1.3.3
  [bd369af6] + Tables v1.12.1
  [72c71f33] + XML v0.3.8
  [49080126] + ZipArchives v2.6.0
  [56f22d72] ~ Artifacts β‡’ v1.11.0
  [ade2ca70] ~ Dates β‡’ v1.11.0
  [a63ad114] ~ Mmap β‡’ v1.11.0
  [de0858da] ~ Printf β‡’ v1.11.0
  [9a3f8284] ~ Random β‡’ v1.11.0
  [cf7118a7] ~ UUIDs β‡’ v1.11.0
  [4ec0a83e] ~ Unicode β‡’ v1.11.0
    Updating `~/work/XLSX.jl/XLSX.jl/Manifest.toml`
  [dce04be8] + ArgCheck v2.5.0
  [6309b1aa] + CodecInflate64 v0.1.3
  [944b1d66] + CodecZlib v0.7.8
  [3da002f7] + ColorTypes v0.12.1
  [5ae59095] + Colors v0.13.1
  [9a962f9c] + DataAPI v1.16.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [53c48c17] + FixedPointNumbers v0.8.5
  [0c81fc1b] + InputBuffers v1.1.1
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [bac558e1] + OrderedCollections v1.8.1
  [aea7be01] + PrecompileTools v1.3.3
  [21216c6a] + Preferences v1.5.0
  [189a3867] + Reexport v1.2.2
  [10745b16] + Statistics v1.11.1
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.12.1
  [3bb67fe8] + TranscodingStreams v0.11.3
  [72c71f33] + XML v0.3.8
  [49080126] + ZipArchives v2.6.0
  [56f22d72] ~ Artifacts β‡’ v1.11.0
  [ade2ca70] ~ Dates β‡’ v1.11.0
  [8f399da3] + Libdl v1.11.0
  [37e2e46d] + LinearAlgebra v1.12.0
  [a63ad114] ~ Mmap β‡’ v1.11.0
  [de0858da] ~ Printf β‡’ v1.11.0
  [9a3f8284] ~ Random β‡’ v1.11.0
  [ea8e919c] + SHA v0.7.0
  [fa267f1f] + TOML v1.0.3
  [cf7118a7] ~ UUIDs β‡’ v1.11.0
  [4ec0a83e] ~ Unicode β‡’ v1.11.0
  [e66e0078] + CompilerSupportLibraries_jll v1.3.0+1
  [4536629a] + OpenBLAS_jll v0.3.29+0
  [83775a58] + Zlib_jll v1.3.1+2
  [8e850b90] + libblastrampoline_jll v5.15.0+0

Just to clarify, I’m making a PR on XLSX.jl, which is a package I don’t own. More specifically, I’m making a minor update on an existing PR of mine which previously passed all tests. I changed a very few lines - mostly (but not all) comments - and now I get this failure.

I will also try to back out these changes one at a time to see if any of them specifically is triggering the failure but, as I said before, I think this process hangs before it even gets to my code.

I tried to explain this in my previous message, but perhaps it wasn’t clear enough: this issue has nothing to do with your code, it happens in all julia nightly jobs. Something changed over the last ~week in either the Julia nightly build, or the macOS environment on GitHub Actions (but somehow only affecting only the nightly version of Julia?), someone will have to investigate it.

1 Like