# Reference tests fail locally but pass on GitHub Actions

**URL:** https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271
**Category:** New to Julia
**Tags:** package
**Created:** [June 27, 2025, 7:31am UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271 "2025-06-27T07:31:41Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![simonsteiger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonsteiger/32/209862_2.png) [@simonsteiger](https://discourse.julialang.org/u/simonsteiger)
#### Post date: [June 27, 2025, 7:31am UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/1 "2025-06-27T07:31:41Z")

</div>

Hi,

I have recently run into a strange problem with reference tests while working on a visualisation package. The tests worked fine but have started failing since I changed the compat for Makie.jl and MCMCChains.jl (this is the affected repo [GitHub - simonsteiger/ChainsMakie.jl: Plot MCMC chains with Makie.jl.](https://github.com/simonsteiger/ChainsMakie.jl)).  
Reversing these changes did not make the issue disappear again unfortunately.

In short, I run into what seems like a dependency issue locally but the tests continue to pass on GitHub Actions (also after clearing the GH Actions cache).

I can reproduce this behaviour in a much simpler test package, see the passing tests in this PR [manually add ref image by simonsteiger · Pull Request #1 · simonsteiger/TestPackage.jl · GitHub](https://github.com/simonsteiger/TestPackage.jl/pull/1). Attempting to run `]test` locally in this repo fails with the following error:

```julia
ERROR: LoadError: InitError: MethodError: no method matching chop(::Bool)
The function `chop` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  chop(::AbstractString; head, tail)
   @ Base strings/util.jl:225

Stacktrace:
  [1] is_sixel_supported(tty::Base.TTY)
    @ Sixel ~/.julia/packages/Sixel/LWLcv/src/Sixel.jl:50
  [2] is_sixel_supported
    @ ~/.julia/packages/Sixel/LWLcv/src/Sixel.jl:49 [inlined]
  [3] __init__ ()
    @ ImageInTerminal ~/.julia/packages/ImageInTerminal/U2jcL/src/ImageInTerminal.jl:159
  [4] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1378
  [5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1366
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Nothing; register::Bool)
    @ Base ./loading.jl:1254
  [7] _include_from_serialized (repeats 2 times)
    @ ./loading.jl:1210 [inlined]
  [8] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
    @ Base ./loading.jl:2057
  [9] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2527
 [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2388
 [11] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [12] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2375
 [14] macro expansion
    @ ./loading.jl:2314 [inlined]
 [15] macro expansion
    @ ./lock.jl:273 [inlined]
 [16] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2271
 [17] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [18] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [19] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2260
 [20] include(fname::String)
    @ Main ./sysimg.jl:38
 [21] top-level scope
    @ ~/.julia/dev/TestPackage.jl/test/runtests.jl:4
 [22] include(fname::String)
    @ Main ./sysimg.jl:38
 [23] top-level scope
    @ none:6
during initialization of module ImageInTerminal
in expression starting at /Users/simonsteiger/.julia/dev/TestPackage.jl/test/run_reference_tests.jl:2
in expression starting at /Users/simonsteiger/.julia/dev/TestPackage.jl/test/runtests.jl:4
ERROR: Package TestPackage errored during testing

```

The error is identical to the error I first ran into in my actual package.

Given these error logs, I had a look at Sixel.jl, and found that a new version for it was released on the day that this issue first appeared. I can’t spot what might be causing this issue in said release though.

I am pretty new to package development, so I initially tried restricting Sixel.jl to `0.1.3` (the recent release is `0.1.4`), but this (understandably) did not work because my package does not directly depend on Sixel.

Any help would be greatly appreciated! 🙂

---

<div class="post-metadata">

### Author: ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)
#### Post date: [June 27, 2025, 8:04am UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/2 "2025-06-27T08:04:57Z")

</div>

So, Sixel.jl has a function `query_terminal` that for some reason returns a `Bool` in your case, and that `Bool` is then passed to `chop`, which is not defined for `Bool` arguments. Looking at [the code of `query_terminal`](https://github.com/JuliaIO/Sixel.jl/blob/6060ffdf8a95b1199a3b0d404143da14e65f94b6/src/terminaltools.jl#L51-L71), I can only think of one reason this might happen: an exception is thrown within the `try` block, and the exception is not of type `TimeoutException`. I think there should be a `rethrow()` call at the end of the `catch` block, but to test it first, can you run the following code (taken from Sixel.jl):

```julia
using Dates

import REPL: Terminals
import Base: TTY

struct TimeoutException <: Exception
    timeout::Float64 # seconds
end

function timeout_call(f::Function, timeout::Real; pollint=0.1)
    start = now()

    t = @task f()
    schedule(t)

    while !istaskdone(t)
        if (now()-start).value >= 1000timeout
            schedule(t, TimeoutException(timeout), error=true)
            sleep(pollint) # wait a while for the task to update its state
            break
        end
        sleep(pollint)
    end

    if t.state == :failed
        throw(t.exception)
    else
        return t.result
    end
end

function with_raw(f, tty::Terminals.TTYTerminal)
    Terminals.raw!(tty, true)
    try
        return f()
    finally
        Terminals.raw!(tty, false)
    end
end

function query_terminal(msg, tty::TTY; timeout=1)
    term = Terminals.TTYTerminal("", stdin, tty, stderr)
    try
        timeout_call(timeout; pollint=timeout/100) do
            with_raw(term) do
                write(tty, msg)
                return @static if Sys.iswindows()
                    response = ""
                    while !endswith(response, 'c')
                        response *= read(stdin, Char)
                    end
                    response
                else
                    transcode(String, readavailable(tty))
                end
            end
        end
    catch e
        e isa TimeoutException && return ""
        rethrow()
    end
end

@info query_terminal("\033[0c", stdout)

```

EDIT: ok, I can reproduce with your testing package: it throws a `MethodError`:

```julia
ERROR: LoadError: InitError: MethodError: no method matching check_open(::IOStream)
The function `check_open` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  check_open(::Base.Filesystem.File)
   @ Base filesystem.jl:199
  check_open(::Union{Base.LibuvServer, Base.LibuvStream})
   @ Base stream.jl:386

Stacktrace:
  [1] timeout_call(f::Sixel.TerminalTools.var"#7#9"{String, Base.TTY, REPL.Terminals.TTYTerminal}, timeout::Int64; pollint::Float64)
    @ Sixel.TerminalTools ~/.julia/packages/Sixel/LWLcv/src/terminaltools.jl:34
  [2] timeout_call
    @ ~/.julia/packages/Sixel/LWLcv/src/terminaltools.jl:18 [inlined]
  [3] query_terminal(msg::String, tty::Base.TTY; timeout::Int64)
    @ Sixel.TerminalTools ~/.julia/packages/Sixel/LWLcv/src/terminaltools.jl:54
  [4] query_terminal
    @ ~/.julia/packages/Sixel/LWLcv/src/terminaltools.jl:51 [inlined]
  [5] is_sixel_supported(tty::Base.TTY)
    @ Sixel ~/.julia/packages/Sixel/LWLcv/src/Sixel.jl:49
  [6] is_sixel_supported
    @ ~/.julia/packages/Sixel/LWLcv/src/Sixel.jl:49 [inlined]
  [7] __init__ ()
    @ ImageInTerminal ~/.julia/packages/ImageInTerminal/U2jcL/src/ImageInTerminal.jl:159
  [8] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1378
  [9] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1366
 [10] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Nothing; register::Bool)
    @ Base ./loading.jl:1254
 [11] _include_from_serialized (repeats 2 times)
    @ ./loading.jl:1210 [inlined]
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
    @ Base ./loading.jl:2057
 [13] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2527
 [14] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2388
 [15] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [16] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [17] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2375
 [18] macro expansion
    @ ./loading.jl:2314 [inlined]
 [19] macro expansion
    @ ./lock.jl:273 [inlined]
 [20] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2271
 [21] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [22] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [23] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2260
 [24] include(fname::String)
    @ Main ./sysimg.jl:38
 [25] top-level scope
    @ /tmp/tmp.YRnS7Y667l/TestPackage.jl/test/runtests.jl:4
 [26] include(fname::String)
    @ Main ./sysimg.jl:38
 [27] top-level scope
    @ none:6

```

---

<div class="post-metadata">

### Author: ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)
#### Post date: [June 27, 2025, 9:22am UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/3 "2025-06-27T09:22:20Z")

</div>

That missing `rethrow` is definitely suboptimal since it silences these errors ([PR created](https://github.com/JuliaIO/Sixel.jl/pull/32)).

Nevertheless, the problem is that when tests are executed, `stdin` is an `IOStream`, for which there is no `check_open` method that is called in `Base.Terminals.raw!`. `Base.Terminals.raw!` was updated in [Don't throw an error in raw! if the stream is closed by Keno · Pull Request #56589 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/56589) and the PR removed the call to `check_open`, so this issue should be fixed in newer Julia versions.

I am not sure why the same problem does not appear on CI though. I am quite confused in general, to be honest 🙂

---

<div class="post-metadata">

### Author: ![simonsteiger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonsteiger/32/209862_2.png) [@simonsteiger](https://discourse.julialang.org/u/simonsteiger)
#### Post date: [June 27, 2025, 12:24pm UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/4 "2025-06-27T12:24:37Z")

</div>

Thank you for having a look at this @barucden and for the explanation! If I understand you correctly, the code should work again in some future Julia release?

Still intriguing with the CI not showing the same error, and the code having worked without issues until so recently.

---

<div class="post-metadata">

### Author: ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)
#### Post date: [June 27, 2025, 1:34pm UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/5 "2025-06-27T13:34:56Z")

</div>

> [@simonsteiger](#):
>
> If I understand you correctly, the code should work again in some future Julia release?

Yes. Julia 1.12 and newer should work for you.

> [@simonsteiger](#):
>
> Still intriguing with the CI not showing the same error, and the code having worked without issues until so recently.

These are two separate issues.

# It worked until recently

Two days ago, Sixel.jl merged a PR ([Fix querying device attributes on Windows by jwortmann · Pull Request #31 · JuliaIO/Sixel.jl · GitHub](https://github.com/JuliaIO/Sixel.jl/pull/31)) that changed the `is_sixel_supported` function. Before that PR, the function looked like this:

```julia
is_sixel_supported(tty) = '4' in query_terminal(...)

```

Function `query_terminal` is assumed to return a `String`, but because of the missing `rethrow` (see my PR linked in my previous post here), `query_terminal` can return `false` in case of an error when interacting with the terminal. Indeed, interacting with the terminal throws en error when running tests locally, and so `query_terminal` returns `false` in that case, and thus, `is_sixel_supported` also returns `false` (because `'4' in false` is `false`).

After the PR was merged two days ago, the function definition changed to

```julia
is_sixel_supported(tty) = "4" in split(chop(query_terminal(...)), ';')

```

but `query_terminal` still returns `false` when running tests localy, and that `false` is then passed to `chop`, which only accepts strings. That’s why you started noticing the error (`MethodError: no method matching chop(::Bool)`) recently.

# It works on CI

I cannot explain this one. When running local tests, `stdin` is an `IOStream` on my computer, in which case `query_terminal` constantly returns `false` (before my PR to Sixel) or throws an error (after my PR to Sixel). I can only imagine that `stdin` is of different type on CI. I don’t know if that’s true or why it should be that way.

---

<div class="post-metadata">

### Author: ![simonsteiger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonsteiger/32/209862_2.png) [@simonsteiger](https://discourse.julialang.org/u/simonsteiger)
#### Post date: [June 27, 2025, 1:41pm UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/6 "2025-06-27T13:41:53Z")

</div>

Thank you once again for the detailed explanation! It’s a great way to learn and I appreciate it a lot, even if it doesn’t solve my issue immediately. (That is not _super_ important anyway 🙂 )

---

<div class="post-metadata">

### Author: ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)
#### Post date: [June 27, 2025, 1:52pm UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/7 "2025-06-27T13:52:00Z")

</div>

You could “fix” it by committing a little type-piracy. If you define

```julia
Base.check_open(io::IOStream) = isopen(io) || throw(ArgumentError("stream closed"))

```

at the beginning of your `runtests.jl` then the error should go away. But it’s quite a dirty solution; don’t push it to your package repository 🙂

---

<div class="post-metadata">

### Author: ![jwortmann](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@jwortmann](https://discourse.julialang.org/u/jwortmann)
#### Post date: [June 28, 2025, 7:18pm UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/8 "2025-06-28T19:18:27Z")

</div>

This bug should be fixed now and a new version v0.1.5 of Sixel.jl with a patch was just released.

By the way, I think your CI was not affected because it is only running on Ubuntu, while the bug should have only happened on Windows.

---

<div class="post-metadata">

### Author: ![simonsteiger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonsteiger/32/209862_2.png) [@simonsteiger](https://discourse.julialang.org/u/simonsteiger)
#### Post date: [June 29, 2025, 7:56am UTC](https://discourse.julialang.org/t/reference-tests-fail-locally-but-pass-on-github-actions/130271/9 "2025-06-29T07:56:13Z")

</div>

Thank you for fixing the bug! The reference tests are running without errors again locally (this was on macOS).