AssertionError: 0 < tid <= length(THREAD_RNGs) during compilation with PackageCompiler

Hey there,
I’m trying to compile a project using the PackageCompiler module. The compilation is part of a conda build process, i.e. I’m trying to make and upload a bioconda package from my project. However, I get the following error during compilation: ERROR: LoadError: LoadError: LoadError: LoadError: AssertionError: 0 < tid <= length(THREAD_RNGs)
See Stacktrace below. I am using the 1.6.2 version of Julia and the 1.6.0 PackageCompiler.

09:06:16 BIOCONDA INFO (OUT) [ Info: PackageCompiler: creating system image object file, this might take a while...
.
09:06:57 BIOCONDA INFO (OUT) ┌ Info: Compiling bit-parallel GC counter for LongSequence{<:NucleicAcidAlphabet}
09:06:57 BIOCONDA INFO (OUT) └ @ BioSequences /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:9
09:06:58 BIOCONDA INFO (OUT) ┌ Info: Compiling bit-parallel mismatch counter for LongSequence{<:NucleicAcidAlphabet}
09:06:58 BIOCONDA INFO (OUT) └ @ BioSequences /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:26
09:06:58 BIOCONDA INFO (OUT) ┌ Info: Compiling bit-parallel match counter for LongSequence{<:NucleicAcidAlphabet}
09:06:58 BIOCONDA INFO (OUT) └ @ BioSequences /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:46
09:06:58 BIOCONDA INFO (OUT) ┌ Info: Compiling bit-parallel ambiguity counter...
09:06:58 BIOCONDA INFO (OUT) └ @ BioSequences /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:73
09:06:58 BIOCONDA INFO (OUT) ┌ Info: 	For a single LongSequence{<:NucleicAcidAlphabet}
09:06:58 BIOCONDA INFO (OUT) └ @ BioSequences /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:74
09:06:58 BIOCONDA INFO (OUT) ┌ Info: 	For a pair of LongSequence{<:NucleicAcidAlphabet}s
09:06:58 BIOCONDA INFO (OUT) └ @ BioSequences /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:88
09:06:58 BIOCONDA INFO (OUT) ┌ Info: Compiling bit-parallel certainty counter for LongSequence{<:NucleicAcidAlphabet}
09:06:58 BIOCONDA INFO (OUT) └ @ BioSequences /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:119
09:06:58 BIOCONDA INFO (OUT) ┌ Info: Compiling bit-parallel gap counter for LongSequence{<:NucleicAcidAlphabet}
09:06:58 BIOCONDA INFO (OUT) └ @ BioSequences /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:140
09:07:01 BIOCONDA INFO (OUT) ERROR: LoadError: LoadError: LoadError: LoadError: AssertionError: 0 < tid <= length(THREAD_RNGs)
09:07:01 BIOCONDA INFO (OUT) Stacktrace:
09:07:01 BIOCONDA INFO (OUT)   [1] _rng_length_assert()
09:07:01 BIOCONDA INFO (OUT)     @ Random /opt/conda/conda-bld/patchwork_1633337321114/_build_env/share/julia/stdlib/v1.6/Random/src/RNGs.jl:381
09:07:01 BIOCONDA INFO (OUT)   [2] default_rng(tid::Int64)
09:07:01 BIOCONDA INFO (OUT)     @ Random /opt/conda/conda-bld/patchwork_1633337321114/_build_env/share/julia/stdlib/v1.6/Random/src/RNGs.jl:372
09:07:01 BIOCONDA INFO (OUT)   [3] default_rng
09:07:01 BIOCONDA INFO (OUT)     @ /opt/conda/conda-bld/patchwork_1633337321114/_build_env/share/julia/stdlib/v1.6/Random/src/RNGs.jl:370 [inlined]
09:07:01 BIOCONDA INFO (OUT)   [4] rand(#unused#::Type{Int64})
09:07:01 BIOCONDA INFO (OUT)     @ Random /opt/conda/conda-bld/patchwork_1633337321114/_build_env/share/julia/stdlib/v1.6/Random/src/Random.jl:259
09:07:01 BIOCONDA INFO (OUT)   [5] top-level scope
09:07:01 BIOCONDA INFO (OUT)     @ ~/.julia/packages/BioSequences/k4j4J/src/longsequences/randseq.jl:11
09:07:01 BIOCONDA INFO (OUT)   [6] include(mod::Module, _path::String)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./Base.jl:386
09:07:01 BIOCONDA INFO (OUT)   [7] include(x::String)
09:07:01 BIOCONDA INFO (OUT)     @ BioSequences ~/.julia/packages/BioSequences/k4j4J/src/BioSequences.jl:8
09:07:01 BIOCONDA INFO (OUT)   [8] top-level scope
09:07:01 BIOCONDA INFO (OUT)     @ ~/.julia/packages/BioSequences/k4j4J/src/BioSequences.jl:284
09:07:01 BIOCONDA INFO (OUT)   [9] include
09:07:01 BIOCONDA INFO (OUT)     @ ./Base.jl:386 [inlined]
09:07:01 BIOCONDA INFO (OUT)  [10] _require(pkg::Base.PkgId)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:1072
09:07:01 BIOCONDA INFO (OUT)  [11] require(uuidkey::Base.PkgId)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:936
09:07:01 BIOCONDA INFO (OUT)  [12] require(into::Module, mod::Symbol)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:923
09:07:01 BIOCONDA INFO (OUT)  [13] include
09:07:01 BIOCONDA INFO (OUT)     @ ./Base.jl:386 [inlined]
09:07:01 BIOCONDA INFO (OUT)  [14] _require(pkg::Base.PkgId)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:1072
09:07:01 BIOCONDA INFO (OUT)  [15] require(uuidkey::Base.PkgId)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:936
09:07:01 BIOCONDA INFO (OUT)  [16] require(into::Module, mod::Symbol)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:923
09:07:01 BIOCONDA INFO (OUT)  [17] include
09:07:01 BIOCONDA INFO (OUT)     @ ./Base.jl:386 [inlined]
09:07:01 BIOCONDA INFO (OUT)  [18] _require(pkg::Base.PkgId)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:1072
09:07:01 BIOCONDA INFO (OUT)  [19] require(uuidkey::Base.PkgId)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:936
09:07:01 BIOCONDA INFO (OUT)  [20] require(into::Module, mod::Symbol)
09:07:01 BIOCONDA INFO (OUT)     @ Base ./loading.jl:923
09:07:01 BIOCONDA INFO (OUT) in expression starting at /home/conda/.julia/packages/BioSequences/k4j4J/src/longsequences/randseq.jl:11
09:07:01 BIOCONDA INFO (OUT) in expression starting at /home/conda/.julia/packages/BioSequences/k4j4J/src/BioSequences.jl:8
09:07:01 BIOCONDA INFO (OUT) in expression starting at /home/conda/.julia/packages/BioAlignments/t4D8A/src/BioAlignments.jl:3
09:07:01 BIOCONDA INFO (OUT) in expression starting at /opt/conda/conda-bld/patchwork_1633337321114/work/src/Patchwork.jl:4
09:07:02 BIOCONDA INFO (OUT) ERROR: LoadError: failed process: Process(`/opt/conda/conda-bld/patchwork_1633337321114/_build_env/bin/julia --color=yes --startup-file=no '--cpu-target=generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)' --sysimage=/tmp/jl_9MfIes/tmp_sys.so --project=/opt/conda/conda-bld/patchwork_1633337321114/work --output-o=/tmp/jl_kN83dR.o -e "Base.reinit_stdio()
09:07:02 BIOCONDA INFO (OUT) @eval Sys BINDIR = ccall(:jl_get_julia_bindir, Any, ())::String
09:07:02 BIOCONDA INFO (OUT) @eval Sys STDLIB = \"/opt/conda/conda-bld/patchwork_1633337321114/_build_env/share/julia/stdlib/v1.6\"
09:07:02 BIOCONDA INFO (OUT) Base.init_load_path()
09:07:02 BIOCONDA INFO (OUT) if isdefined(Base, :init_active_project)
09:07:02 BIOCONDA INFO (OUT)     Base.init_active_project()
09:07:02 BIOCONDA INFO (OUT) end
09:07:02 BIOCONDA INFO (OUT) Base.init_depot_path()
09:07:02 BIOCONDA INFO (OUT) import Patchwork
09:07:02 BIOCONDA INFO (OUT) # This @eval prevents symbols from being put into Main
09:07:02 BIOCONDA INFO (OUT) @eval Module() begin
09:07:02 BIOCONDA INFO (OUT)     using Base.Meta
09:07:02 BIOCONDA INFO (OUT)     PrecompileStagingArea = Module()
09:07:02 BIOCONDA INFO (OUT)     for (_pkgid, _mod) in Base.loaded_modules
09:07:02 BIOCONDA INFO (OUT)         if !(_pkgid.name in (\"Main\", \"Core\", \"Base\"))
09:07:02 BIOCONDA INFO (OUT)             eval(PrecompileStagingArea, :(const \$(Symbol(_mod)) = \$_mod))
09:07:02 BIOCONDA INFO (OUT)         end
09:07:02 BIOCONDA INFO (OUT)     end
09:07:02 BIOCONDA INFO (OUT)     precompile_files = String[
09:07:02 BIOCONDA INFO (OUT)         \"/tmp/jl_VZQiK8\"
09:07:02 BIOCONDA INFO (OUT)         \"/opt/conda/conda-bld/patchwork_1633337321114/work/src/precompiled.jl\"
09:07:02 BIOCONDA INFO (OUT)     ]
09:07:02 BIOCONDA INFO (OUT)     for file in precompile_files, statement in eachline(file)
09:07:02 BIOCONDA INFO (OUT)         try
09:07:02 BIOCONDA INFO (OUT)             # println(statement)
09:07:02 BIOCONDA INFO (OUT)             # This is taken from https://github.com/JuliaLang/julia/blob/2c9e051c460dd9700e6814c8e49cc1f119ed8b41/contrib/generate_precompile.jl#L375-L393
09:07:02 BIOCONDA INFO (OUT)             ps = Meta.parse(statement)
09:07:02 BIOCONDA INFO (OUT)             isexpr(ps, :call) || continue
09:07:02 BIOCONDA INFO (OUT)             popfirst!(ps.args) # precompile(...)
09:07:02 BIOCONDA INFO (OUT)             ps.head = :tuple
09:07:02 BIOCONDA INFO (OUT)             l = ps.args[end]
09:07:02 BIOCONDA INFO (OUT)             if (isexpr(l, :tuple) || isexpr(l, :curly)) && length(l.args) > 0 # Tuple{...} or (...)
09:07:02 BIOCONDA INFO (OUT)                 # XXX: precompile doesn't currently handle overloaded Vararg arguments very well.
09:07:02 BIOCONDA INFO (OUT)                 # Replacing N with a large number works around it.
09:07:02 BIOCONDA INFO (OUT)                 l = l.args[end]
09:07:02 BIOCONDA INFO (OUT)                 if isexpr(l, :curly) && length(l.args) == 2 && l.args[1] === :Vararg # Vararg{T}
09:07:02 BIOCONDA INFO (OUT)                     push!(l.args, 100) # form Vararg{T, 100} instead
09:07:02 BIOCONDA INFO (OUT)                 end
09:07:02 BIOCONDA INFO (OUT)             end
09:07:02 BIOCONDA INFO (OUT)             # println(ps)
09:07:02 BIOCONDA INFO (OUT)             ps = Core.eval(PrecompileStagingArea, ps)
09:07:02 BIOCONDA INFO (OUT)             # XXX: precompile doesn't currently handle overloaded nospecialize arguments very well.
09:07:02 BIOCONDA INFO (OUT)             # Skipping them avoids the warning.
09:07:02 BIOCONDA INFO (OUT)             ms = length(ps) == 1 ? Base._methods_by_ftype(ps[1], 1, Base.get_world_counter()) : Base.methods(ps...)
09:07:02 BIOCONDA INFO (OUT)             ms isa Vector || continue
09:07:02 BIOCONDA INFO (OUT)             precompile(ps...)
09:07:02 BIOCONDA INFO (OUT)         catch e
09:07:02 BIOCONDA INFO (OUT)             # See julia issue #28808
09:07:02 BIOCONDA INFO (OUT)             @debug \"failed to execute \$statement\"
09:07:02 BIOCONDA INFO (OUT)         end
09:07:02 BIOCONDA INFO (OUT)     end
09:07:02 BIOCONDA INFO (OUT) end # module
09:07:02 BIOCONDA INFO (OUT) Base.@ccallable function julia_main()::Cint
09:07:02 BIOCONDA INFO (OUT)     try
09:07:02 BIOCONDA INFO (OUT)         Patchwork.julia_main()
09:07:02 BIOCONDA INFO (OUT)     catch
09:07:02 BIOCONDA INFO (OUT)         Core.print(\"julia_main() threw an unhandled exception\")
09:07:02 BIOCONDA INFO (OUT)         return 1
09:07:02 BIOCONDA INFO (OUT)     end
09:07:02 BIOCONDA INFO (OUT) end
09:07:02 BIOCONDA INFO (OUT) empty!(LOAD_PATH)
09:07:02 BIOCONDA INFO (OUT) empty!(DEPOT_PATH)
09:07:02 BIOCONDA INFO (OUT) "`, ProcessExited(1)) [1]
09:07:02 BIOCONDA INFO (OUT) 
09:07:02 BIOCONDA INFO (OUT) Stacktrace:
09:07:02 BIOCONDA INFO (OUT)   [1] pipeline_error
09:07:02 BIOCONDA INFO (OUT)     @ ./process.jl:525 [inlined]
09:07:02 BIOCONDA INFO (OUT)   [2] run(::Cmd; wait::Bool)
09:07:02 BIOCONDA INFO (OUT)     @ Base ./process.jl:440
09:07:02 BIOCONDA INFO (OUT)   [3] run
09:07:02 BIOCONDA INFO (OUT)     @ ./process.jl:438 [inlined]
09:07:02 BIOCONDA INFO (OUT)   [4] create_sysimg_object_file(object_file::String, packages::Vector{String}; project::String, base_sysimage::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, cpu_target::String, script::Nothing, isapp::Bool, sysimage_build_args::Cmd)
09:07:02 BIOCONDA INFO (OUT)     @ PackageCompiler ~/.julia/packages/PackageCompiler/r8Xcy/src/PackageCompiler.jl:371
09:07:02 BIOCONDA INFO (OUT)   [5] create_sysimage(packages::Vector{String}; sysimage_path::String, project::String, precompile_execution_file::Vector{String}, precompile_statements_file::String, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, base_sysimage::String, isapp::Bool, julia_init_c_file::Nothing, version::Nothing, compat_level::String, soname::String, cpu_target::String, script::Nothing, sysimage_build_args::Cmd)
09:07:02 BIOCONDA INFO (OUT)     @ PackageCompiler ~/.julia/packages/PackageCompiler/r8Xcy/src/PackageCompiler.jl:538
09:07:02 BIOCONDA INFO (OUT)   [6] (::PackageCompiler.var"#14#16"{Bool, Nothing, Nothing, String, Cmd, String, Vector{String}, String, Bool, Bool, String, String, String, String, Bool})()
09:07:02 BIOCONDA INFO (OUT)     @ PackageCompiler ~/.julia/packages/PackageCompiler/r8Xcy/src/PackageCompiler.jl:978
09:07:02 BIOCONDA INFO (OUT)   [7] cd(f::PackageCompiler.var"#14#16"{Bool, Nothing, Nothing, String, Cmd, String, Vector{String}, String, Bool, Bool, String, String, String, String, Bool}, dir::String)
09:07:02 BIOCONDA INFO (OUT)     @ Base.Filesystem ./file.jl:106
09:07:02 BIOCONDA INFO (OUT)   [8] _create_app(package_dir::String, dest_dir::String, name::String, precompile_execution_file::Vector{String}, precompile_statements_file::String, incremental::Bool, filter_stdlibs::Bool, audit::Bool, force::Bool, cpu_target::String; library_only::Bool, c_driver_program::String, julia_init_c_file::Nothing, header_files::Vector{String}, version::Nothing
, compat_level::String, include_lazy_artifacts::Bool, sysimage_build_args::Cmd)
09:07:02 BIOCONDA INFO (OUT)     @ PackageCompiler ~/.julia/packages/PackageCompiler/r8Xcy/src/PackageCompiler.jl:968
09:07:02 BIOCONDA INFO (OUT)   [9] #create_app#11
09:07:02 BIOCONDA INFO (OUT)     @ ~/.julia/packages/PackageCompiler/r8Xcy/src/PackageCompiler.jl:771 [inlined]
09:07:02 BIOCONDA INFO (OUT)  [10] top-level scope
09:07:02 BIOCONDA INFO (OUT)     @ /opt/conda/conda-bld/patchwork_1633337321114/work/src/compile.jl:21
09:07:02 BIOCONDA INFO (OUT) in expression starting at /opt/conda/conda-bld/patchwork_1633337321114/work/src/compile.jl:21
09:07:04 BIOCONDA INFO (OUT) Traceback (most recent call last):
09:07:04 BIOCONDA INFO (OUT)   File "/opt/conda/bin/conda-build", line 11, in <module>
09:07:04 BIOCONDA INFO (OUT)     sys.exit(main())
09:07:04 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 481, in main
09:07:04 BIOCONDA INFO (OUT)     execute(sys.argv[1:])
09:07:04 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 472, in execute
09:07:04 BIOCONDA INFO (OUT)     verify=args.verify, variants=args.variants, cache_dir=args.cache_dir)
09:07:04 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/api.py", line 195, in build
09:07:04 BIOCONDA INFO (OUT)     variants=variants
09:07:04 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 3073, in build_tree
09:07:04 BIOCONDA INFO (OUT)     notest=notest,
09:07:04 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 2192, in build
09:07:04 BIOCONDA INFO (OUT)     cwd=src_dir, stats=build_stats)
09:07:04 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/utils.py", line 412, in check_call_env
09:07:04 BIOCONDA INFO (OUT)     return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
09:07:04 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/utils.py", line 392, in _func_defaulting_env_to_os_environ
09:07:04 BIOCONDA INFO (OUT)     raise subprocess.CalledProcessError(proc.returncode, _args)
09:07:04 BIOCONDA INFO (OUT) subprocess.CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/opt/conda/conda-bld/patchwork_1633337321114/work/conda_build.sh']' returned non-zero exit status 1.
09:07:16 BIOCONDA ERROR COMMAND FAILED (exited with 1): docker run -t --net host --rm -v /tmp/tmpqr8l0uzh/build_script.bash:/opt/build_script.bash -v /home/circleci/project/miniconda/conda-bld/:/opt/host-conda-bld -v /home/circleci/project/recipes/patchwork:/opt/recipe -e LANG=en_US.UTF-8 -e HOST_USER_ID=1001 quay.io/bioconda/bioconda-utils-build-env-cos7:0.17.8 /bin/bash /opt/build_script.bash

09:07:16 BIOCONDA ERROR BUILD FAILED recipes/patchwork
09:07:16 BIOCONDA INFO (COMMAND) conda build purge
09:07:18 BIOCONDA ERROR BUILD SUMMARY: of 1 recipes, 1 failed and 0 were skipped. Details of recipes and environments follow.
09:07:18 BIOCONDA ERROR BUILD SUMMARY: FAILED recipe recipes/patchwork


Exited with code exit status 1

I think the command that produces the error is PackageCompiler.create_app(projectdirectory, outdirectory; precompile_statements_file = precompiled, force = true).
Could someone help me please? Thanks!

1 Like

The package is calling rand in toplevel

But the rng is not fully functional in the process that creates the sysimage. However, we precompile things first to avoid this problem. I will try check what is going on.

1 Like

Okay, thanks, that would be brilliant.

I just ran into this problem today, same assertion error. Thanks for looking into it!

initialize RNGs in sysimage process by KristofferC · Pull Request #579 · JuliaLang/PackageCompiler.jl · GitHub should fix it but I am still unclear why this is needed.

Also, I cannot reproduce this locally. More information about this would be good because it means an assumption I have is not completely correct.

For someone who has this error, could you run with ENV["JULIA_DEBUG"] = "loading" before creating the sysimage?

Hmm. When I ran my conda build test again today, it worked fine. Creating the sysimage locally (independent from conda) didn’t error either. Should I still send the debug output?

Only useful when it errors. Anyway, I made a new release of PackageCompiler that should work around this even in the strange case. It should be available soon.

1 Like

Updated to v1.7.0 and I no longer have that error, thank you again!

Can you reliably reproduce it on PackageCompiler 1.6? In that case, could you explain how?

So I rolled back to 1.6 and tried both locally and on the cluster and the assertion error returned. For reproducibility, I added XAM.jl to the MyApp example (added to dev’d package, only included “using XAM” in src/MyApp.jl) and it started failing with the same assertion error both locally and on the cluster.

I’m dealing with another issue or two (PackageCompiler "create_app" function failing on slurm cluster: "error: undefined reference to 'ceil'") but I don’t think that changes the conclusion that the PackageCompiler v1.6.0 issue can be reproduced with that XAM.jl package?

Please let me know if you need any more details!

Thanks, I can repro! Great.

1 Like

Are you creating an app or a sysimage? If you are creating an app, could you pass incremental=false to check if that works around it?

I am trying to create an app, but unfortunately for neither the error: undefined reference to 'ceil' errors on the cluster nor the seg fault error locally (induced by trying to run post-compilation) does incremental=false change anything.

Thank you again for your help and all ears for any other suggestions. Going to try dismantling my app or adding parts that might break the example app and see if anything changes. Will let you know if I find any possible causes.

Using PackageCompiler 1.7.2, I am still running into precisely this error when attempting to generate a sysimage containing Makie packages. The offending dependency appears to be TiffImages. I did not have this issue when precompiling Makie for my previous Julia installation (I recently upgraded from Julia 1.6.2 to 1.6.3). Is there any workaround/fix I can do here?

Please provide full details of how you create the sysimage so it can possibly be reproduced.