Cryptic error messages, no idea what to do

General Usage
1

Hi,

I’ve been developing a Julia module to be used on multiple machines. Recently, I noticed the module wouldn’t compile on one of the machines (my macbook, actually). It still runs fine on my HPC cluster. Been a while I hadn’t tested it on my macbook and several modifications were made since then, so I have absolutely no idea what triggered this.

I’m getting the following error messages. Can anyone help?

I’ve tried deleting the Manifest.toml file but that didn’t help. The error messages mention TestExt as missing but when I try adding that package it says package not found. Don’t have so much experience with Julia so I’m a bit clueless here. It seems like my module compiles fine, but as soon as save() is called everything goes bad.

Thanks all.

julia> using DrWatson

julia> @quickactivate :FlowComplexity
  Activating project at `~/Documents/academic/3-research/projects/2023 spatial-topology/flow-complexity`

julia> include("queue.jl")
Precompiling CSVFiles
  ✓ TranscodingStreams
  16 dependencies successfully precompiled in 22 seconds. 61 already precompiled.
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version
[ Info: Precompiling CSVFiles [5d742f6a-9f54-50ce-8119-2520741973ca]
┌ Warning: Module CodecZlib with build ID fafbfcfd-62e0-4f8a-0000-8b202c50c609 is missing from the cache.
│ This may mean CodecZlib [944b1d66-785c-5afd-91f1-9de20f533193] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1942
[ Info: Skipping precompilation since __precompile__(false). Importing CSVFiles [5d742f6a-9f54-50ce-8119-2520741973ca].
[ Info: Precompiling TextParse [e0df1984-e451-5cb5-8b61-797a481e67e3]
┌ Warning: Module CodecZlib with build ID fafbfcfd-62e0-4f8a-0000-8b202c50c609 is missing from the cache.
│ This may mean CodecZlib [944b1d66-785c-5afd-91f1-9de20f533193] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1942
[ Info: Skipping precompilation since __precompile__(false). Importing TextParse [e0df1984-e451-5cb5-8b61-797a481e67e3].
[ Info: Precompiling DoubleFloats [497a8b3b-efae-58df-a0af-a86822472b78]
┌ Warning: Module SpecialFunctions with build ID fafbfcfd-02e8-4981-0000-8b1b976bbfef is missing from the cache.
│ This may mean SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1942
[ Info: Skipping precompilation since __precompile__(false). Importing DoubleFloats [497a8b3b-efae-58df-a0af-a86822472b78].
[ Info: Precompiling ConstructionBaseStaticArraysExt [8497ba20-d017-5d93-8a79-2639523b7219]
┌ Warning: Module StaticArrays with build ID fafbfcfd-c3b5-dda9-0000-8b1dad1456f0 is missing from the cache.
│ This may mean StaticArrays [90137ffa-7385-5640-81b9-e52037218182] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1942
[ Info: Skipping precompilation since __precompile__(false). Importing ConstructionBaseStaticArraysExt [8497ba20-d017-5d93-8a79-2639523b7219].
[ Info: Precompiling HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]
┌ Warning: Module JLLWrappers with build ID fafbfcfd-f1ca-ec9a-0000-8b1bae0bd77d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1942
[ Info: Skipping precompilation since __precompile__(false). Importing HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3].
[ Info: Precompiling OpenSSL [4d8831e6-92b7-49fb-bdf8-b643e874388c]
┌ Warning: Module JLLWrappers with build ID fafbfcfd-f1ca-ec9a-0000-8b1bae0bd77d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1942
[ Info: Skipping precompilation since __precompile__(false). Importing OpenSSL [4d8831e6-92b7-49fb-bdf8-b643e874388c].
[ Info: Precompiling OpenSSL_jll [458c3c95-2e84-50aa-8efc-19380b2a3a95]
┌ Warning: Module JLLWrappers with build ID fafbfcfd-f1ca-ec9a-0000-8b1bae0bd77d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1942
[ Info: Skipping precompilation since __precompile__(false). Importing OpenSSL_jll [458c3c95-2e84-50aa-8efc-19380b2a3a95].
┌ Error: Error during loading of extension TestExt of TranscodingStreams, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    ArgumentError: Package TestExt [29286dec-c893-52d1-88fe-e07e79eb4405] is required but does not seem to be installed:
│     - Run `Pkg.instantiate()` to install all recorded dependencies.
│
│    Stacktrace:
│      [1] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1920
│      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1806
│      [3] #invoke_in_world#3
│        @ Base ./essentials.jl:921 [inlined]
│      [4] invoke_in_world
│        @ Base ./essentials.jl:918 [inlined]
│      [5] _require_prelocked
│        @ Base ./loading.jl:1797 [inlined]
│      [6] _require_prelocked
│        @ Base ./loading.jl:1796 [inlined]
│      [7] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1289
│      [8] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1324
│      [9] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1158
│     [10] _tryrequire_from_serialized(modkey::Base.PkgId, path::String, ocachepath::String, sourcepath::String, depmods::Vector{Any})
│        @ Base ./loading.jl:1481
│     [11] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
│        @ Base ./loading.jl:1568
│     [12] _require(pkg::Base.PkgId, env::String)
│        @ Base ./loading.jl:1932
│     [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1806
│     [14] #invoke_in_world#3
│        @ Base ./essentials.jl:921 [inlined]
│     [15] invoke_in_world
│        @ Base ./essentials.jl:918 [inlined]
│     [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1797
│     [17] macro expansion
│        @ Base ./loading.jl:1784 [inlined]
│     [18] macro expansion
│        @ Base ./lock.jl:267 [inlined]
│     [19] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1747
│     [20] #invoke_in_world#3
│        @ Base ./essentials.jl:921 [inlined]
│     [21] invoke_in_world
│        @ Base ./essentials.jl:918 [inlined]
│     [22] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1740
│     [23] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [24] include(x::String)
│        @ HTTP ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:1
│     [25] top-level scope
│        @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:35
│     [26] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [27] _require(pkg::Base.PkgId, env::String)
│        @ Base ./loading.jl:2008
│     [28] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1806
│     [29] #invoke_in_world#3
│        @ Base ./essentials.jl:921 [inlined]
│     [30] invoke_in_world
│        @ Base ./essentials.jl:918 [inlined]
│     [31] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1797
│     [32] macro expansion
│        @ Base ./loading.jl:1784 [inlined]
│     [33] macro expansion
│        @ Base ./lock.jl:267 [inlined]
│     [34] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1747
│     [35] #invoke_in_world#3
│        @ Base ./essentials.jl:921 [inlined]
│     [36] invoke_in_world
│        @ Base ./essentials.jl:918 [inlined]
│     [37] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1740
│     [38] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [39] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:2008
│     [40] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1806
│     [41] #invoke_in_world#3
│        @ Base ./essentials.jl:921 [inlined]
│     [42] invoke_in_world
│        @ Base ./essentials.jl:918 [inlined]
│     [43] _require_prelocked
│        @ Base ./loading.jl:1797 [inlined]
│     [44] _require_prelocked
│        @ Base ./loading.jl:1796 [inlined]
│     [45] macro expansion
│        @ Base ./lock.jl:267 [inlined]
│     [46] require(uuidkey::Base.PkgId)
│        @ Base ./loading.jl:1791
│     [47] #34
│        @ FileIO ~/.julia/packages/FileIO/xOKyx/src/loadsave.jl:203 [inlined]
│     [48] lock(f::FileIO.var"#34#35"{Base.PkgId}, l::ReentrantLock)
│        @ Base ./lock.jl:229
│     [49] action(call::Symbol, libraries::Vector{Union{Base.PkgId, Module}}, file::FileIO.Formatted, args::DataFrame; options::@Kwargs{})
│        @ FileIO ~/.julia/packages/FileIO/xOKyx/src/loadsave.jl:203
│     [50] action
│        @ ~/.julia/packages/FileIO/xOKyx/src/loadsave.jl:196 [inlined]
│     [51] action
│        @ ~/.julia/packages/FileIO/xOKyx/src/loadsave.jl:185 [inlined]
│     [52] save(file::String, args::DataFrame; options::@Kwargs{})
│        @ FileIO ~/.julia/packages/FileIO/xOKyx/src/loadsave.jl:129
│     [53] save(file::String, args::DataFrame)
│        @ FileIO ~/.julia/packages/FileIO/xOKyx/src/loadsave.jl:125
│     [54] top-level scope
│        @ ~/Documents/academic/3-research/projects/2023 spatial-topology/flow-complexity/milestones/05_mass-exploration/queue.jl:47
│     [55] include(fname::String)
│        @ Base.MainInclude ./client.jl:489
│     [56] top-level scope
│        @ REPL[3]:1
│     [57] eval
│        @ Core ./boot.jl:385 [inlined]
│     [58] eval_user_input(ast::Any, backend::REPL.REPLBackend, mod::Module)
│        @ REPL ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:150
│     [59] repl_backend_loop(backend::REPL.REPLBackend, get_module::Function)
│        @ REPL ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:246
│     [60] start_repl_backend(backend::REPL.REPLBackend, consumer::Any; get_module::Function)
│        @ REPL ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:231
│     [61] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool, backend::Any)
│        @ REPL ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:389
│     [62] run_repl(repl::REPL.AbstractREPL, consumer::Any)
│        @ REPL ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:375
│     [63] (::Base.var"#1013#1015"{Bool, Bool, Bool})(REPL::Module)
│        @ Base ./client.jl:432
│     [64] #invokelatest#2
│        @ Base ./essentials.jl:887 [inlined]
│     [65] invokelatest
│        @ Base ./essentials.jl:884 [inlined]
│     [66] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
│        @ Base ./client.jl:416
│     [67] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:333
└ @ Base loading.jl:1295
Precompiling DistributionsTestExt
  3 dependencies successfully precompiled in 9 seconds. 41 already precompiled.
[ Info: Precompiling DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019]
┌ Warning: Module Distributions with build ID fafbfcfd-5ff9-10c0-0000-8b1b78772e28 is missing from the cache.
│ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1942
[ Info: Skipping precompilation since __precompile__(false). Importing DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019].

julia>
2

You’re not giving us much to work with to help you. Can’t you create a reproducible example?

3

I don’t even know what’s causing all these errors. It works perfectly fine on several other machines, and used to (also) work fine on my macbook. I wouldn’t even know where to start to create a minimal example. Isn’t there something else I can try?

4

This looks similar to Error during loading of extension TestExt of TranscodingStreams, use `Base.retry_load_extensions()` to retry · Issue #234 · JuliaIO/TranscodingStreams.jl · GitHub

First of all, to get an environment to reproducable work on multiple machines I run julia with JULIA_LOAD_PATH="@" julia --project=<add path to project here> --startup-file=no This ensures the particular machines startup file and global environment cannot have any effects.

5

The other thing you can do is update the packages in both the global environment and local environment. The newest version of TranscodingStreams.jl doesn’t have a TestExt, so you must be using a old version somewhere.

6

Interesting. Tried this, no error message was shown. I’ve pinpointed this to using --project=\<path\>. Whenever I use --project my module works fine, whereas if I just launch julia then use DrWatson’s @quickactivate I get all these error messages.

I guess I’ll try to dig into what the @quickactivate macro does to see if I can pinpoint further what’s causing this. But at this point I’m tempted to just reinstall Julia and see if that fixes the issue… I’ve tried deleting everything in ~/.julia/compiled/v1.10 but that didn’t change anything.

7

What does running the following show?

julia -e 'using Pkg; Pkg.status(PKGMODE_MANIFEST)'
8 
>>> julia -e 'using Pkg; Pkg.status(PKGMODE_MANIFEST)'
Status `~/.julia/environments/v1.11/Manifest.toml` (empty manifest)

Otherwise, reinstalling Julia did not help. In fact, it’s worse now: even when using --project=path I’m getting these cryptic error messages all over again. I’m totally puzzled now.

9

It looks like you changed from Julia v1.10 to v1.11. If you have juliaup, you can do juliaup add 1.10 and then JULIA_LOAD_PATH="@" julia +1.10 --project=<add path to project here> --startup-file=no. You may have to re instantiate the environment to download the packages.

10

@nhz2 I think I’ve found a solution: you had suggested I update the TranscodingStreams.jl package in my environment (don’t even know what’s it used for, anyway), so I tried to update in the Pkg REPL, and both my Project.toml and Manifest.toml files were updated and now everything compiles.

I really want to get to the bottom of this so I’ll restore the .toml files from the git repo and try to reproduce. I still no idea what caused this, or why was it working on some machines and not on my macbook.

11

Alright I have reproduced this:

  • reinstalled Julia once more
  • tried to Pkg.instantiate() my module, got a bunch of error messages
  • when trying to julia <somefile.jl> (where this calls @quickactivate on my module) it can’t compile and I get more error messages

This seems to have fixed it:

  • start julia with --project=<module path>
  • get into the Pkg REPL using ], and type update—all packages get updated, my Manifest.toml file reflects this, and now everything works fine

I’ve tried to do update TranscodingStreams specifically but it wouldn’t update because of some unsatisfied requirement. But after running update I’m seeing TranscodingStreams was updated from 0.11.2 to 0.11.3.

I have no idea what caused this. One hypothesis is that this might have to do with having to downgrade the version of DrWatson used by my module, as the newer version required using a version of JLD2 which had issues. I know from the Manifest.toml file that TranscodingStreams is required by JLD2. I don’t remember exactly how I downgraded (remember, I’m still new to Julia) but I may have “forced” the downgrade or something. I guess that might have created package version discrepancies somewhere down the line, and that for some reason this only created problems now and on specific machines only.

12

Note that these “error messages” are in fact just warnings, everything should just work if you ignore them! I admit it’s not made clear by how Julia outputs them…
The specific TestExt issue (seems applicable to stdlibs more generally) is a known reported issue in Julia, just turned out nontrivial to fix – so many packages did various workarounds for that.

13

Hmm, right. What’s confusing is that there’s not only Warnings but also Errors:

Error: Error during loading of extension TestExt

The code looked like it indeed ran, but seeing 150 lines of warnings/error messages is pretty concerning! Especially for people new to Julia like me.

14

I’m not sure the errors are harmless here (though without a reproducible example, it is hard to say) Ref: Package not installed locally can be accessed by "using", but incorrectly · Issue #55071 · JuliaLang/julia · GitHub

When you run using DrWatson after starting Julia without the --project it loads the version of JLD2 that is in your ~/.julia/environments/v1.11/Manifest.toml.
Then when you run @quickactivate :FlowComplexity you will still be using the version of JLD2 in the global environment, not the version of JLD2 that you have in your “FlowComplexity” Manifest.toml. Similar reproducibility issues can happen if you load any packages in the startup.jl file (that’s why I add --startup-file=no in addition to --project=... if reproducibility is important)

1 Like
15

Is there any way to achieve the same behavior than --startup-file=no through scripting (i.e., in a .jl file) ?

Maybe one solution would be to include it in the file’s shebang:

#!/usr/bin/env julia --startup-file=no
...

In any case, that could well be why I had these errors on one machine and not the other: I had multiple versions of package X installed on e.g. my machine, and when I used @quickactivate I still had these versions active instead of the ones in the Manifest.toml, whereas this wasn’t the case on e.g. the HPC cluster.

Also: I’m guessing VSC starts the REPL using something like julia --project=<path>, which would also explain why I wasn’t getting any error message inside it?

16

Alright so correct me if I’m wrong, but actually I would simply need to do something like this:

using Pkg
Pkg.add(name="DrWatson", version="2.17.0")
using DrWatson
@quickactivate :FlowComplexity

to ensure reproducibility? And where obviously the DrWatson version should match the one required by the project.

This way I’d be 100% sure that all package loaded have versions corresponding to the project manifest?

edit: this creates Manifest files all over the place so a better solution could be to pin the specific version of DrWatson in the REPL:

Pkg.add(name="DrWatson", version="2.17.0")
Pkg.pin(name="DrWatson", version="2.17.0")
Pkg.status()
17

If you start Julia with the --project=... option, do you need to run @quickactivate? I haven’t used that macro before, so I’m not sure what it is needed for.

18

https://modernjuliaworkflows.org/writing/#environments

Has some good notes on how to use environments in VSCode.

1 Like
19

No, @quickactivate searches and loads the module recursively, which I found more useful than --project=...

For context, I’m using the module for scientific calculations and have several subfolders such as AAA/111/..., AAA/222/..., BBB/111, and so on (with even more depth at times) that make use of the module found in the parent folder. This makes it a bit tiresome to use --project which is why I preferred using @quickactivate.

I’m using VSC for development but oftentimes will call the julia executable from a shell (either locally or on the HPC cluster).

I think making sure I Pkg.pin the right version of DrWatson as I suggested earlier in my edit would ensure complete reproducibility.

20

Just delete your startup file, making the option unnecessary? Weird question TBH.