Bug in LinearAlgebra? How to debug a standard library

Is this known?

[91133] signal 11 (2): Segmentation fault: 11
in expression starting at /Users/pkrysl/ACol.jl/runsim.jl:38
gc_alloc_map_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/./gc.h:360 [inlined]
gc_sweep_page at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:1549 [inlined]
gc_sweep_pool_page at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:1574
gc_sweep_prescan at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:1640
gc_sweep_wake_all at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:1661 [inlined]
gc_sweep_pool at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:1806 [inlined]
_jl_gc_collect at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:3645
ijl_gc_collect at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:3893
maybe_collect at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:926 [inlined]
jl_gc_pool_alloc_inner at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:1319 [inlined]
jl_gc_pool_alloc_noinline at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gc.c:1386 [inlined]
jl_gc_alloc_ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/./julia_internal.h:523
_new_genericmemory_ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/genericmemory.c:56
GenericMemory at ./boot.jl:516 [inlined]
new_as_memoryref at ./boot.jl:535 [inlined]
Array at ./boot.jl:582 [inlined]
similar at ./array.jl:370 [inlined]
AbstractArray at ./array.jl:627
convert at ./abstractarray.jl:18 [inlined]
QRPivoted at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/LinearAlgebra/src/qr.jl:451
Factorization at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/LinearAlgebra/src/qr.jl:453 [inlined]
ldiv at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/LinearAlgebra/src/LinearAlgebra.jl:643
\ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/LinearAlgebra/src/LinearAlgebra.jl:631 [inlined]
\ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/LinearAlgebra/src/generic.jl:1134

Mac M2 Ultra, Julia 1.11.7.

Segfaults in GC are most frequently due to unrelated corruption (typically an out-of-bounds write). Try running with check bounds=yes … or better yet, use rr and go back in time to see what wrote to the corrupted address (but that’s hard and Linux-only).

3 Likes

Some notes about debugging garbage collector bugs with rr: How I debug issues with Julia GC

2 Likes

Bounds checking getting turned off was instructive.

ERROR: LoadError: BoundsError: attempt to access 12582-element Vector{ComplexF32} at index [11995693056]
Stacktrace:
  [1] throw_boundserror(A::Vector{ComplexF32}, I::Tuple{Int64})
    @ Base ./essentials.jl:14
  [2] setindex!
    @ ./array.jl:986 [inlined]
  [3] ldiv!(A::LinearAlgebra.QRPivoted{ComplexF32, Matrix{ComplexF32}, Vector{ComplexF32}, Vector{Int64}}, B::Matrix{ComplexF32}, rcond::Float32)
    @ LinearAlgebra ~/julia-1.11.7/share/julia/stdlib/v1.11/LinearAlgebra/src/qr.jl:607
  [4] ldiv!
    @ ~/julia-1.11.7/share/julia/stdlib/v1.11/LinearAlgebra/src/qr.jl:620 [inlined]
  [5] ldiv(F::LinearAlgebra.QRPivoted{ComplexF32, Matrix{ComplexF32}, Vector{ComplexF32}, Vector{Int64}}, B::Matrix{ComplexF32})
    @ LinearAlgebra ~/julia-1.11.7/share/julia/stdlib/v1.11/LinearAlgebra/src/LinearAlgebra.jl:656
  [6] \
    @ ~/julia-1.11.7/share/julia/stdlib/v1.11/LinearAlgebra/src/LinearAlgebra.jl:631 [inlined]
  [7] \(A::Matrix{ComplexF32}, B::Matrix{ComplexF32})
    @ LinearAlgebra ~/julia-1.11.7/share/julia/stdlib/v1.11/LinearAlgebra/src/generic.jl:1134

But I still don’t know what happened. I want to dev LinearAlgebra in order to put some
debugging prints in there, but it doesn’t seem to work. What do I do? I don’t think I had to debug
a standard library before.
This doesn’t seem to have done it:

Can you maybe share the code you’re executing that results in the error?

1 Like

I’m afraid I can’t do that. Proprietary data and code. I don’t have an MWE (yet).

1 Like

Does anyone know how to force Julia to pick up a modified version of the LinearAlgebra?
I see
image
yet the prints are not coming. I think the code still uses the precompiled library.
How do I change that? precompile does not work.
image

That is the line:
image
The pivot is garbage.

1 Like
3 Likes

The easiest approach for simple debugging is often to just monkey patch the method you’re working on. No custom sysimage or Julia build needed.

julia> @eval LinearAlgebra function foo(bar, baz)
           # code copied from the real LinearAlgebra.foo
           println("what on earth happens here? x=$x, y=$y")
           # the rest of the real LinearAlgebra.foo
       end
7 Likes

Could Revise.track(#=standard library name=#) work as a way to do those @evals? I wouldn’t want to alter the original source files and don’t actually know the proper way to use that, I’m just curious if LinearAlgebra being part of the sysimage would affect Revise as it affects dev.

Is there a way to view the LinearAlgebra code as it is in Julia 1.11? If you just go to the GitHub repo, the line numbers don’t match up with the error messages, so it’s changed relative to what OP was using.

1 Like

In GitHub, you can just find the correct 1.11 release tag and browse the code that way:

Or you can launch the version of Julia you want and then use @less or @edit and it will open the correct code. (And you can look at the file path to further browse locally.)

2 Likes

Oh, is this not the right place? GitHub - JuliaLang/LinearAlgebra.jl: Julia Linear Algebra standard library

1 Like

In recent versions of Julia (v1.12 and up, maybe?), LinearAlgebra.jl was separated into a standalone repo. This line in the julia repo points to the relevant Git object in the LinearAlgebra.jl repo.

So append the hash string to, for example, https://github.com/JuliaLang/LinearAlgebra.jl/commit/ to get a Github URL. Like this: