ERROR: KeyError: key "SparseArrays" not found

New to Julia
1

I’m new to Julia, I had a project that was using ControlSystems and SparseArrays packages. After the weekend I came back to the office, lauched a test, and nothing was working. Basically it says that I cannot compile ControlSystems (why does it need to compile it again?) because the Key SparseArrays is missing.
I spent 4h trying to debug the issue, and I’ve tried everything I could find, including all Pkg.x() commands (all failing with the same error “ERROR: KeyError: key “SparseArrays” not found”), and also deleting all local Julia Package, compiled, and environment files and restarting, just to find myself at the same point.

I’m kinda of lost, and a bit upset (I wanted to try Julia so much to move away from the shitty Python packages creating issues all the time).

Stack trace of me trying to recompile ControlSystems after I reset everything

ERROR: LoadError: Failed to precompile ControlSystems [a6e380b2-a6ca-5380-bf3e-84a91bcd477e] to "/path/.julia/compiled/v1.10/ControlSystems/jl_VwYKFv".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [24] top-level scope
    @ /path/test_sim.jl:9

Stacktrace of a Pkg.resolve()

ERROR: KeyError: key "SparseArrays" not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:498 [inlined]
  [2] status_ext_info(pkg::Pkg.Types.PackageSpec, env::Pkg.Types.EnvCache)
    @ Pkg.Operations /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2214
  [3] print_status(env::Pkg.Types.EnvCache, old_env::Pkg.Types.EnvCache, registries::Vector{…}, header::Symbol, uuids::Vector{…}, names::Vector{…}; manifest::Bool, diff::Bool, ignore_indent::Bool, outdated::Bool, extensions::Bool, io::Base.TTY, mode::Pkg.Types.PackageMode, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2299
  [4] print_status
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2241 [inlined]
  [5] status(env::Pkg.Types.EnvCache, registries::Vector{…}, pkgs::Vector{…}; header::Symbol, mode::Pkg.Types.PackageMode, git_diff::Bool, env_diff::Pkg.Types.EnvCache, ignore_indent::Bool, io::Base.TTY, outdated::Bool, extensions::Bool, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2475
  [6] status (repeats 2 times)
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2443 [inlined]
  [7] #show_update#167
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2439 [inlined]
  [8] up(ctx::Pkg.Types.Context, pkgs::Vector{…}, level::Pkg.Types.UpgradeLevel; skip_writing_project::Bool, preserve::Nothing)
    @ Pkg.Operations /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1545
  [9] up
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1520 [inlined]
 [10] up(ctx::Pkg.Types.Context, pkgs::Vector{…}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, preserve::Nothing, update_registry::Bool, skip_writing_project::Bool, kwargs::@Kwargs{})
    @ Pkg.API /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/API.jl:351
 [11] up
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/API.jl:326 [inlined]
 [12] up
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/API.jl:164 [inlined]
 [13] #resolve#143
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/API.jl:357 [inlined]
 [14] resolve
    @ /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/API.jl:356 [inlined]
 [15] resolve(; io::Base.TTY, kwargs::@Kwargs{})
    @ Pkg.API /tools/foss/julia/1.10.4/x86_64/gen_linux/share/julia/stdlib/v1.10/Pkg/src/API.jl:355
 [16] top-level scope
    @ REPL[5]:1
Some type information was truncated. Use `show(err)` to see complete types.
2

Welcome to the forum!

Can you try to activate a temporary environment and add only ControlSystems.jl? You can do it as follows:

julia> using Pkg

julia> Pkg.activate(temp=true)
  Activating new project at `/tmp/jl_ZjU2Cv`

julia> Pkg.add("ControlSystems")

Does it also fail to precompile, or does it succeed? You should be able to run

julia> using ControlSystems
3

Hi! Thank for the answer. I’ve already tried and gave me the same error.
I forgot to mention that ofc “using SparseArrays” works like a charm and is basically the only command that does not give me errors

4

I think you are getting the same error as reported here: `status` can error when the RHS in `[extensions]` does not map to a weak dep · Issue #3947 · JuliaLang/Pkg.jl · GitHub

But I am not sure why does it happen in your case.

Does that mean that you tried removing (or rather renaming temporarily) ~/.julia and it still did not help? What happens if you check ] status in your package environment?

julia> using Pkg

julia> Pkg.status()
5

Yep, I tried removing the .julia and nothiing changed.
The Pkg.status() returns

Project NAME v0.1.0
Status `/path/NAME/Project.toml`
  [a6e380b2] ControlSystems v1.11.2
  [634d3b9d] DrWatson v2.18.0
  [2f01184e] SparseArrays v1.10.0

The initial project had way more packages, but I’ve been cleaning and resetting all morning.

Regarding the link you provided me, I did not get it very well (consider I’m new). Should I go into the Manifest.toml to search for SparseArraysExt?

6

Not exactly. The error you posted is raised when an extension has a dependency that is not defined as a (weak) dependency (relevant part of the docs: 5. Creating Packages · Pkg.jl).

The piece of code which raises the error has been modified since v1.10 to provide a more informative message. Can you try to add the package with julia 1.11 by any chance?

1 Like
7

Sadly I cannot, since I’m using a shared cluster.
Do you have any suggestions?
Why Friday was working and now not? Can I install an older version of ControlSystem and SparseArrays?

8

I have no idea why your project stopped working. The dependencies do not change unless you explicitly ask for it (or change Project.toml manually).

Uh, I am out of my depth here. Let’s try to collect more information about the packages and their extensions:

julia> using Pkg

julia> Pkg.status(extensions=true, mode=PKGMODE_MANIFEST)

This lists all the dependencies of your package together with their extensions and the dependencies of the extensions.

Edit: FWIW, I can add ControlSystems to a fresh environment without any issues in julia 1.10. Since you see the error in a fresh environment with a clean ~/.julia, then I can only imagine something’s wrong with the Julia installation on your cluster. Perhaps something has changed over the weekend? Maybe you could install juliaup (GitHub - JuliaLang/juliaup: Julia installer and version multiplexer) and install julia to your home directory?

1 Like
9

Yeah.
I’ve decided to “rip off the patch” and I installed a personal julia (1.11 since I was at it). Now it works without any issues.
Thanks anyway for your support

1 Like
10

I also have this problem in CI with Julia 1.10 on Windows and macOS. Ubuntu is fine, and Julia 1.11 is also working on all platforms.

11

What patch version of v1.10? There were some issues with extensions, that had been fixed in latest patches.

12

Thank you! That’s it! We were running CI on Windows and macOS only with Julia versions ‘min’ and ‘1’, so the earliest supported version for the package and the latest release. I now realized that ‘min’ is using 1.10.0, which doesn’t really make sense to test, so we now changed it to ‘1.10’, which solved the problem.

1 Like