Reference tests fail locally but pass on GitHub Actions

New to Julia
1

Hi,

I have recently run into a strange problem with reference tests while working on a visualisation package. The tests worked fine but have started failing since I changed the compat for Makie.jl and MCMCChains.jl (this is the affected repo GitHub - simonsteiger/ChainsMakie.jl: Plot MCMC chains with Makie.jl.).
Reversing these changes did not make the issue disappear again unfortunately.

In short, I run into what seems like a dependency issue locally but the tests continue to pass on GitHub Actions (also after clearing the GH Actions cache).

I can reproduce this behaviour in a much simpler test package, see the passing tests in this PR manually add ref image by simonsteiger · Pull Request #1 · simonsteiger/TestPackage.jl · GitHub. Attempting to run ]test locally in this repo fails with the following error:

ERROR: LoadError: InitError: MethodError: no method matching chop(::Bool)
The function `chop` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  chop(::AbstractString; head, tail)
   @ Base strings/util.jl:225

Stacktrace:
  [1] is_sixel_supported(tty::Base.TTY)
    @ Sixel ~/.julia/packages/Sixel/LWLcv/src/Sixel.jl:50
  [2] is_sixel_supported
    @ ~/.julia/packages/Sixel/LWLcv/src/Sixel.jl:49 [inlined]
  [3] __init__()
    @ ImageInTerminal ~/.julia/packages/ImageInTerminal/U2jcL/src/ImageInTerminal.jl:159
  [4] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1378
  [5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1366
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Nothing; register::Bool)
    @ Base ./loading.jl:1254
  [7] _include_from_serialized (repeats 2 times)
    @ ./loading.jl:1210 [inlined]
  [8] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
    @ Base ./loading.jl:2057
  [9] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2527
 [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2388
 [11] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [12] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2375
 [14] macro expansion
    @ ./loading.jl:2314 [inlined]
 [15] macro expansion
    @ ./lock.jl:273 [inlined]
 [16] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2271
 [17] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [18] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [19] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2260
 [20] include(fname::String)
    @ Main ./sysimg.jl:38
 [21] top-level scope
    @ ~/.julia/dev/TestPackage.jl/test/runtests.jl:4
 [22] include(fname::String)
    @ Main ./sysimg.jl:38
 [23] top-level scope
    @ none:6
during initialization of module ImageInTerminal
in expression starting at /Users/simonsteiger/.julia/dev/TestPackage.jl/test/run_reference_tests.jl:2
in expression starting at /Users/simonsteiger/.julia/dev/TestPackage.jl/test/runtests.jl:4
ERROR: Package TestPackage errored during testing

The error is identical to the error I first ran into in my actual package.

Given these error logs, I had a look at Sixel.jl, and found that a new version for it was released on the day that this issue first appeared. I can’t spot what might be causing this issue in said release though.

I am pretty new to package development, so I initially tried restricting Sixel.jl to 0.1.3 (the recent release is 0.1.4), but this (understandably) did not work because my package does not directly depend on Sixel.

Any help would be greatly appreciated! :slight_smile:

2

So, Sixel.jl has a function query_terminal that for some reason returns a Bool in your case, and that Bool is then passed to chop, which is not defined for Bool arguments. Looking at the code of query_terminal, I can only think of one reason this might happen: an exception is thrown within the try block, and the exception is not of type TimeoutException. I think there should be a rethrow() call at the end of the catch block, but to test it first, can you run the following code (taken from Sixel.jl):

using Dates

import REPL: Terminals
import Base: TTY

struct TimeoutException <: Exception
    timeout::Float64 # seconds
end

function timeout_call(f::Function, timeout::Real; pollint=0.1)
    start = now()

    t = @task f()
    schedule(t)

    while !istaskdone(t)
        if (now()-start).value >= 1000timeout
            schedule(t, TimeoutException(timeout), error=true)
            sleep(pollint) # wait a while for the task to update its state
            break
        end
        sleep(pollint)
    end

    if t.state == :failed
        throw(t.exception)
    else
        return t.result
    end
end

function with_raw(f, tty::Terminals.TTYTerminal)
    Terminals.raw!(tty, true)
    try
        return f()
    finally
        Terminals.raw!(tty, false)
    end
end

function query_terminal(msg, tty::TTY; timeout=1)
    term = Terminals.TTYTerminal("", stdin, tty, stderr)
    try
        timeout_call(timeout; pollint=timeout/100) do
            with_raw(term) do
                write(tty, msg)
                return @static if Sys.iswindows()
                    response = ""
                    while !endswith(response, 'c')
                        response *= read(stdin, Char)
                    end
                    response
                else
                    transcode(String, readavailable(tty))
                end
            end
        end
    catch e
        e isa TimeoutException && return ""
        rethrow()
    end
end

@info query_terminal("\033[0c", stdout)

EDIT: ok, I can reproduce with your testing package: it throws a MethodError:

ERROR: LoadError: InitError: MethodError: no method matching check_open(::IOStream)
The function `check_open` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  check_open(::Base.Filesystem.File)
   @ Base filesystem.jl:199
  check_open(::Union{Base.LibuvServer, Base.LibuvStream})
   @ Base stream.jl:386

Stacktrace:
  [1] timeout_call(f::Sixel.TerminalTools.var"#7#9"{String, Base.TTY, REPL.Terminals.TTYTerminal}, timeout::Int64; pollint::Float64)
    @ Sixel.TerminalTools ~/.julia/packages/Sixel/LWLcv/src/terminaltools.jl:34
  [2] timeout_call
    @ ~/.julia/packages/Sixel/LWLcv/src/terminaltools.jl:18 [inlined]
  [3] query_terminal(msg::String, tty::Base.TTY; timeout::Int64)
    @ Sixel.TerminalTools ~/.julia/packages/Sixel/LWLcv/src/terminaltools.jl:54
  [4] query_terminal
    @ ~/.julia/packages/Sixel/LWLcv/src/terminaltools.jl:51 [inlined]
  [5] is_sixel_supported(tty::Base.TTY)
    @ Sixel ~/.julia/packages/Sixel/LWLcv/src/Sixel.jl:49
  [6] is_sixel_supported
    @ ~/.julia/packages/Sixel/LWLcv/src/Sixel.jl:49 [inlined]
  [7] __init__()
    @ ImageInTerminal ~/.julia/packages/ImageInTerminal/U2jcL/src/ImageInTerminal.jl:159
  [8] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1378
  [9] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1366
 [10] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Nothing; register::Bool)
    @ Base ./loading.jl:1254
 [11] _include_from_serialized (repeats 2 times)
    @ ./loading.jl:1210 [inlined]
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
    @ Base ./loading.jl:2057
 [13] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2527
 [14] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2388
 [15] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [16] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [17] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2375
 [18] macro expansion
    @ ./loading.jl:2314 [inlined]
 [19] macro expansion
    @ ./lock.jl:273 [inlined]
 [20] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2271
 [21] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [22] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [23] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2260
 [24] include(fname::String)
    @ Main ./sysimg.jl:38
 [25] top-level scope
    @ /tmp/tmp.YRnS7Y667l/TestPackage.jl/test/runtests.jl:4
 [26] include(fname::String)
    @ Main ./sysimg.jl:38
 [27] top-level scope
    @ none:6
1 Like
3

That missing rethrow is definitely suboptimal since it silences these errors (PR created).

Nevertheless, the problem is that when tests are executed, stdin is an IOStream, for which there is no check_open method that is called in Base.Terminals.raw!. Base.Terminals.raw! was updated in Don't throw an error in raw! if the stream is closed by Keno · Pull Request #56589 · JuliaLang/julia · GitHub and the PR removed the call to check_open, so this issue should be fixed in newer Julia versions.

I am not sure why the same problem does not appear on CI though. I am quite confused in general, to be honest :slight_smile:

4

Thank you for having a look at this @barucden and for the explanation! If I understand you correctly, the code should work again in some future Julia release?

Still intriguing with the CI not showing the same error, and the code having worked without issues until so recently.

5

Yes. Julia 1.12 and newer should work for you.

These are two separate issues.

It worked until recently

Two days ago, Sixel.jl merged a PR (Fix querying device attributes on Windows by jwortmann · Pull Request #31 · JuliaIO/Sixel.jl · GitHub) that changed the is_sixel_supported function. Before that PR, the function looked like this:

is_sixel_supported(tty) = '4' in query_terminal(...)

Function query_terminal is assumed to return a String, but because of the missing rethrow (see my PR linked in my previous post here), query_terminal can return false in case of an error when interacting with the terminal. Indeed, interacting with the terminal throws en error when running tests locally, and so query_terminal returns false in that case, and thus, is_sixel_supported also returns false (because '4' in false is false).

After the PR was merged two days ago, the function definition changed to

is_sixel_supported(tty) = "4" in split(chop(query_terminal(...)), ';')

but query_terminal still returns false when running tests localy, and that false is then passed to chop, which only accepts strings. That’s why you started noticing the error (MethodError: no method matching chop(::Bool)) recently.

It works on CI

I cannot explain this one. When running local tests, stdin is an IOStream on my computer, in which case query_terminal constantly returns false (before my PR to Sixel) or throws an error (after my PR to Sixel). I can only imagine that stdin is of different type on CI. I don’t know if that’s true or why it should be that way.

1 Like
6

Thank you once again for the detailed explanation! It’s a great way to learn and I appreciate it a lot, even if it doesn’t solve my issue immediately. (That is not super important anyway :slight_smile: )

7

You could “fix” it by committing a little type-piracy. If you define

Base.check_open(io::IOStream) = isopen(io) || throw(ArgumentError("stream closed"))

at the beginning of your runtests.jl then the error should go away. But it’s quite a dirty solution; don’t push it to your package repository :slight_smile:

1 Like