LinearSolveCUDAExt fails to compile

Specific Domains Optimization (Mathematical)
1

As a dependency of DiffEqGPU (version 3.8,0), it fails to compile on Julia 1.11.6 (Linux).

(Test) pkg> precompile
Precompiling project...
  ✗ LinearSolve → LinearSolveCUDAExt
  33 dependencies successfully precompiled in 131 seconds. 583 already precompiled.
  1 dependency had output during precompilation:
┌ Test
│  ERROR: LoadError: UndefVarError: `LinearVerbosity` not defined in `LinearSolveCUDAExt`
│  Stacktrace:
│   [1] top-level scope
│     @ ~/.julia/packages/LinearSolve/fHamo/ext/LinearSolveCUDAExt.jl:52
│   [2] include
│     @ ./Base.jl:562 [inlined]
│   [3] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│     @ Base ./loading.jl:2881
│   [4] top-level scope
│     @ stdin:6
│  in expression starting at /home/sob/.julia/packages/LinearSolve/fHamo/ext/LinearSolveCUDAExt.jl:1
│  in expression starting at stdin:6
│  ┌ Error: Error during loading of extension LinearSolveCUDAExt of LinearSolve, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Failed to precompile LinearSolveCUDAExt [e24d4dde-ed20-5ee7-b465-f1dd73f9b6ba] to "/home/sob/.julia/compiled/v1.11/LinearSolveCUDAExt/jl_7Ajqnz".
│  │    Stacktrace:
│  │      [1] error(s::String)
│  │        @ Base ./error.jl:35
│  │      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, loadable_exts::Vector{Base.PkgId})
│  │        @ Base ./loading.jl:3174
│  │      [3] (::Base.var"#1110#1111"{Base.PkgId})()
│  │        @ Base ./loading.jl:2579

Anybody getting the same error?

1 Like
2

I’m getting the same error in two different machines.

Julia Version 1.11.6
Commit 9615af0f269 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 7 1700 Eight-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver1)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

Julia Version 1.11.6
Commit 9615af0f269 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver1)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
1 Like
3

Sorry, there was a patch where this caused an issue with the CUDA extension. It should not be fixed on LinearSolve v3.34.0

1 Like
4

Works now thank you @ChrisRackauckas

5

Still getting a warning that init_cacheval() is overwritten though.

6

Can you share that warning?

7

New empty project upon calling add CUDA DiffEqGPU (Linux, Julia 1.11.6)

Precompiling project...
  ? LinearSolve → LinearSolveCUDAExt
  1 dependency successfully precompiled in 3 seconds. 227 already precompiled.
  1 dependencies failed but may be precompilable after restarting julia
  1 dependency had output during precompilation:
┌ LinearSolve → LinearSolveCUDAExt
│  WARNING: Method definition init_cacheval(LinearSolve.CudaOffloadLUFactorization, Any, Any, Any, Any, Any, Int64, Any, Any, LinearSolve.LinearVerbosity{T} where T, LinearSolve.OperatorAssumptions{T} where T) in module LinearSolve at /home/sob/.julia/packages/LinearSolve/507yZ/src/factorization.jl:1213 overwritten in module LinearSolveCUDAExt at /home/sob/.julia/packages/LinearSolve/507yZ/ext/LinearSolveCUDAExt.jl:55.
│  ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
└  
 
(Blah) pkg>
8

Alright we got our GPU CI machines fixed. Try LinearSolve.jl v3.36.0. That should be good now :crossed_fingers:

9

All clear now thanks @ChrisRackauckas

1 Like
10

More issues in the same package @ChrisRackauckas

julia> using Test
Precompiling LinearSolveCUDAExt...
Info Given LinearSolveCUDAExt was explicitly requested, output will be shown live 
WARNING: Method definition usecuda() in module LinearSolve at /home/sob/.julia/packages/LinearSolve/IJofQ/src/LinearSolve.jl:440 overwritten in module LinearSolveCUDAExt at /home/sob/.julia/packages/LinearSolve/IJofQ/ext/LinearSolveCUDAExt.jl:14.
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
  ? LinearSolve → LinearSolveCUDAExt
[ Info: Precompiling LinearSolveCUDAExt [e24d4dde-ed20-5ee7-b465-f1dd73f9b6ba] (cache misses: incompatible header (8))
WARNING: Method definition usecuda() in module LinearSolve at /home/sob/.julia/packages/LinearSolve/IJofQ/src/LinearSolve.jl:440 overwritten in module LinearSolveCUDAExt at /home/sob/.julia/packages/LinearSolve/IJofQ/ext/LinearSolveCUDAExt.jl:14.
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
┌ Info: Skipping precompilation due to precompilable error. Importing LinearSolveCUDAExt [e24d4dde-ed20-5ee7-b465-f1dd73f9b6ba].
└   exception = Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
 
julia>
11

Thanks for the error message. Fixed in Fix precompilation of usecuda, usemetal, and useblis by ChrisRackauckas · Pull Request #792 · SciML/LinearSolve.jl · GitHub

12

Attempting to update my project now has me downgrade a couple packages including LinearSolve:

(Test) pkg> update
    Updating registry at `~/.julia/registries/General.toml`
   Installed NonlinearSolveSpectralMethods ─ v1.5.0
   Installed OptimizationOptimJL ─────────── v0.4.5
   Installed BracketingNonlinearSolve ────── v1.5.0
   Installed NonlinearSolveQuasiNewton ───── v1.10.0
   Installed SimpleNonlinearSolve ────────── v2.9.0
   Installed TimeZones ───────────────────── v1.22.1
   Installed NonlinearSolve ──────────────── v4.12.0
   Installed LinearSolve ─────────────────── v3.28.0
   Installed NonlinearSolveFirstOrder ────── v1.9.0
   Installed NonlinearSolveBase ──────────── v2.0.0
   Installed Distributions ───────────────── v0.25.121
   Installed OptimizationBase ────────────── v2.12.0
   Installed Optimization ────────────────── v4.8.0
    Updating `~/workspace/Test/Project.toml`
  [31c24e10] ↑ Distributions v0.25.120 ⇒ v0.25.121
  [7f7a1694] ↑ Optimization v4.7.0 ⇒ v4.8.0
  [36348300] ↑ OptimizationOptimJL v0.4.4 ⇒ v0.4.5
  [f269a46b] ↑ TimeZones v1.22.0 ⇒ v1.22.1
    Updating `~/workspace/Test/Manifest.toml`
  [70df07ce] ↑ BracketingNonlinearSolve v1.4.0 ⇒ v1.5.0
  [31c24e10] ↑ Distributions v0.25.120 ⇒ v0.25.121
⌃ [7ed4a6bd] ↓ LinearSolve v3.40.2 ⇒ v3.28.0
  [8913a72c] ↑ NonlinearSolve v4.11.0 ⇒ v4.12.0
  [be0214bd] ↑ NonlinearSolveBase v1.16.1 ⇒ v2.0.0
  [5959db7a] ↑ NonlinearSolveFirstOrder v1.8.0 ⇒ v1.9.0
  [9a2c21bd] ↑ NonlinearSolveQuasiNewton v1.9.0 ⇒ v1.10.0
  [26075421] ↑ NonlinearSolveSpectralMethods v1.4.0 ⇒ v1.5.0
  [7f7a1694] ↑ Optimization v4.7.0 ⇒ v4.8.0
  [bca83a33] ↑ OptimizationBase v2.11.0 ⇒ v2.12.0
  [36348300] ↑ OptimizationOptimJL v0.4.4 ⇒ v0.4.5
  [727e6d20] ↑ SimpleNonlinearSolve v2.8.0 ⇒ v2.9.0
⌃ [9672c7b4] ↓ SteadyStateDiffEq v2.5.0 ⇒ v2.4.1
  [f269a46b] ↑ TimeZones v1.22.0 ⇒ v1.22.1
        Info Packages marked with ⌃ have new versions available and may be upgradable.
Precompiling project...
  ✗ SteadyStateDiffEq
  ✗ DifferentialEquations
  ✗ Test
  75 dependencies successfully precompiled in 113 seconds. 550 already precompiled.
  3 dependencies errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`
 
(Test) pkg>

And the root cause seems to be:

ERROR: LoadError: UndefVarError: `AbstractNonlinearTerminationMode` not defined in `SteadyStateDiffEq`

EDIT:

A blank new project with only DifferentialEquations and CUDA compiles fine (including the recent LinearSolve. Meaning the my real world project has a package that is forcing an older version of LinearSolve which errors, but a more recent version shouldn’t.

13

So @ChrisRackauckas it seems SteadyStateDiffEq is held/dowgraded (at v2.4.1, instead of v2.5.0) by a package in the DifferentialEquations or Optimization tree.

I was able to recover a broken environment by doing the following:

  • rm DifferentialEquations
  • add SteadyStateDiffEq (to have explicitely in Project, goes to v2.5.0)
  • add DifferentialEquations (didn’t downgrade SteadyStateDiffEq… however if I were to remove SteadyStateDiffEq from Project, up would downgrade it as a dep to DifferentialEquations)
14

Try with the new SteadyStateDiffEq.jl. It looks like I forgot to do that release with the crucial patch which changes it to using NonlinearSolveBase instead of DiffEqBase

15

The newer package still gets downgraded unless explicitly in Project.toml

(Test) pkg> update
    Updating registry at `~/.julia/registries/General.toml`
  No Changes to `~/workspace/Test/Project.toml`
    Updating `~/workspace/Test/Manifest.toml`
  [70df07ce] ↑ BracketingNonlinearSolve v1.4.0 ⇒ v1.5.0
  [8913a72c] ↑ NonlinearSolve v4.11.0 ⇒ v4.12.0
  [be0214bd] ↑ NonlinearSolveBase v1.16.1 ⇒ v2.0.0
  [5959db7a] ↑ NonlinearSolveFirstOrder v1.8.0 ⇒ v1.9.0
  [9a2c21bd] ↑ NonlinearSolveQuasiNewton v1.9.0 ⇒ v1.10.0
  [26075421] ↑ NonlinearSolveSpectralMethods v1.4.0 ⇒ v1.5.0
  [727e6d20] ↑ SimpleNonlinearSolve v2.8.0 ⇒ v2.9.0
⌃ [9672c7b4] ↓ SteadyStateDiffEq v2.6.0 ⇒ v2.4.1
        Info Packages marked with ⌃ have new versions available and may be upgradable.
16

Check why

17

This is where my knowledge of Julia package management is the limiting factor.

However, I have narrowed it down to the following MWE:

  1. create empty project
  2. add DifferentialEquations
  3. update

What I observe on multiple machines is:

  1. will install DifferentialEquations v7.16.1 and its dependency SteadyStateDiffEq v2.6.0
  2. will downgrade SteadyStateDiffEq v2.6.0 ⇒ v2.4.1 (and
    ForwardDiff v1.2.1 ⇒ v0.10.39)
18

]why SteadyStateDiffEq

19

As per my MWE, it’s there because of DifferentialEquations (sole package in Project).

Additional note on my MWE: using julia 1.12 rc3, it’s necessary to add DiffEqGPU to step 2) to reproduce.

20

Ahh I found the issue. It was a NonlinearSolveBase bound in SteadyStateDiffEq that put things into an odd state. I just created a new release of that. I think we’re good now, :crossed_fingers: sorry for the troubles.