Design of a function which takes ownership of memory, and Julia memory functions (:jl_malloc etc)

Internals & Design
1

I need to implement a function which takes a Julia array (vector is sufficient) and passes ownership to a C function from SuiteSparse:GraphBLAS. This will be a long post so please bear with me.

I want to implement the function GxB_Vector_import_Full which imports a dense vector from Julia to the C library. Importantly the dense Julia vector should no longer be owned/visible to Julia.

Here is the definition of the function:

GB_PUBLIC
GrB_Info GxB_Vector_import_Full // import a full vector
(
    GrB_Vector *v,      // handle of vector to create
    GrB_Type type,      // type of vector to create
    GrB_Index n,        // vector length
    void **vx,          // values, vx_size >= nvals(v) * (type size)
                        // or vx_size >= (type size), if iso is true
    GrB_Index vx_size,  // size of vx in bytes
    bool iso,           // if true, v is iso
    const GrB_Descriptor desc
) ;

This is the docstring above the C function:

The semantics of import/export/pack/unpack are the same as the “move
constructor” in C++. On import, the user provides a set of arrays that have
been previously allocated via the ANSI C malloc function. The arrays define
the content of the matrix or vector. Unlike GrB_*_build, the GraphBLAS
library then takes ownership of the user’s input arrays and may either (a)
incorporate them into its internal data structure for the new GrB_Matrix or
GrB_Vector, potentially creating the GrB_Matrix or GrB_Vector in constant
time with no memory copying performed, or (b) if the library does not
support the import format directly, then it may convert the input to its
internal format, and then free the user’s input arrays. GraphBLAS may also
choose to use a mix of the two strategies. In either case, the input arrays
are no longer “owned” by the user application. If A is a GrB_Matrix created
by an import/pack, the user input arrays are freed no later than GrB_free
(&A), and may be freed earlier, at the discretion of the GraphBLAS library.
The data structure of the GrB_Matrix and GrB_Vector remain opaque.

As I understand it, with input from Keno, it’s not trivial to do this directly. Instead I implemented it as a copy with the following Julia function:

function importdensevec(
    n::Integer, v::Vector{T};
    desc::Descriptor = DEFAULTDESC, iso = false
) where {T}
    w = Ref{libgb.GrB_Vector}()
    n = libgb.GrB_Index(n)
    vsize = libgb.GrB_Index(sizeof(v))

    vx = Ptr{T}(Libc.malloc(vsize))
    unsafe_copyto!(vx, pointer(v), length(v))
    libgb.GxB_Vector_import_Full(
        w,
        toGBType(T),
        n,
        Ref{Ptr{Cvoid}}(vx),
        vsize,
        iso,
        desc
    )
    return GBVector{T}(w[])
end

GrB_Index is just a UInt64.

The problem: This works fine, although I believe it slowly leaks memory. It becomes an issue if I pass GraphBLAS Julia’s memory functions:
libgb.GxB_init(libgb.GrB_NONBLOCKING, cglobal(:jl_malloc), cglobal(:jl_calloc), cglobal(:jl_realloc), cglobal(:jl_free), true).

This is very useful so that Julia’s GC will respond to memory pressure from GraphBLAS. However, finalize on any GBVector will segfault with what is probably a double free.

Relevant sections of code are:

  1. The Julia import function: https://github.com/JuliaSparse/SuiteSparseGraphBLAS.jl/blob/e82a1190a65c4bae0c900d71248060c206baaf3e/src/import.jl#L131
  2. The finalizer: https://github.com/JuliaSparse/SuiteSparseGraphBLAS.jl/blob/e82a1190a65c4bae0c900d71248060c206baaf3e/src/types.jl#L89
  3. The free function called by that finalizer: https://github.com/JuliaSparse/SuiteSparseGraphBLAS.jl/blob/e82a1190a65c4bae0c900d71248060c206baaf3e/src/lib/LibGraphBLAS.jl#L759

Are there any obvious causes of this issue? This probably isn’t enough info to answer directly, so please let me know what additional info is needed.

Some notes:

  • Nothing bad happens if GraphBLAS is given Julia’s memory functions, and all memory is managed by GraphBLAS. That is if I create a v = GBVector{Float64}(100); v[1] = 1; finalize(v) nothing bad happens. It’s only when I use the above import function that segfaults occur.

Here’s the segfault

julia> v = GBVector(rand(10000))
10000x1 GraphBLAS double vector, full by col
  10000 entries, memory: 78.4 KB

    (1,1)    0.551737
    (2,1)    0.545864
    (3,1)    0.499952
    (4,1)    0.288739
    (5,1)    0.941688
    (6,1)    0.220175
    (7,1)    0.642798
    (8,1)    0.486546
    (9,1)    0.263924
    (10,1)    0.396567
    (11,1)    0.392586
    (12,1)    0.352326
    (13,1)    0.160226
    (14,1)    0.498261
    (15,1)    0.10208
    (16,1)    0.704522
    (17,1)    0.572655
    (18,1)    0.216401
    (19,1)    0.647443
    (20,1)    0.106365
    (21,1)    0.205428
    (22,1)    0.350109
    (23,1)    0.0265592
    (24,1)    0.454546
    (25,1)    0.0456463
    (26,1)    0.578271
    (27,1)    0.235998
    (28,1)    0.499136
    (29,1)    0.315318
    ...


julia> v = nothing

julia> GC.gc()
Freeing a vector
munmap_chunk(): invalid pointer

signal (6): Aborted
in expression starting at REPL[5]:1
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f7934894507)
unknown function (ip: 0x7f793489ac19)
unknown function (ip: 0x7f793489b183)
jl_gc_counted_free_with_size at /buildworker/worker/package_linux64/build/src/gc.c:3386
GB_free_memory at /home/will/.julia/artifacts/1f93b9028928b30c16345d26d889adc8815ac7df/lib/libgraphblas.so (unknown line)
GB_dealloc_memory at /home/will/.julia/artifacts/1f93b9028928b30c16345d26d889adc8815ac7df/lib/libgraphblas.so (unknown line)
GB_bix_free at /home/will/.julia/artifacts/1f93b9028928b30c16345d26d889adc8815ac7df/lib/libgraphblas.so (unknown line)
GB_phbix_free at /home/will/.julia/artifacts/1f93b9028928b30c16345d26d889adc8815ac7df/lib/libgraphblas.so (unknown line)
GB_Matrix_free at /home/will/.julia/artifacts/1f93b9028928b30c16345d26d889adc8815ac7df/lib/libgraphblas.so (unknown line)
GrB_Vector_free at /home/will/.julia/artifacts/1f93b9028928b30c16345d26d889adc8815ac7df/lib/libgraphblas.so (unknown line)
GrB_Vector_free at /home/will/.julia/dev/SuiteSparseGraphBLAS.jl/src/lib/LibGraphBLAS.jl:761
f at /home/will/.julia/dev/SuiteSparseGraphBLAS.jl/src/types.jl:89
unknown function (ip: 0x7f78d2a3cf44)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:367
run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394
jl_gc_collect at /buildworker/worker/package_linux64/build/src/gc.c:3259
gc at ./gcutils.jl:94 [inlined]
gc at ./gcutils.jl:94
unknown function (ip: 0x7f78d2a3c4fc)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:139
repl_backend_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:200
start_repl_backend at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:185
#run_repl#42 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:317
run_repl at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:305
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
#874 at ./client.jl:387
jfptr_YY.874_23032.clone_1 at /home/will/julia-1.6.2/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:714
#invokelatest#2 at ./essentials.jl:708 [inlined]
invokelatest at ./essentials.jl:706 [inlined]
run_main_repl at ./client.jl:372
exec_options at ./client.jl:302
_start at ./client.jl:485
jfptr__start_34281.clone_1 at /home/will/julia-1.6.2/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:560
repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at julia (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4007d8)
Allocations: 10696794 (Pool: 10693171; Big: 3623); GC: 12
Aborted
2

Is the issue that GraphBLAS is trying to free memory created by Libc.malloc using :jl_free rather than Libc.free? I don’t think you can use different memory management functions right?

Indeed I believe this is the solution. Instead of using Libc.malloc we need to use ccall(:jl_malloc, Ptr{T}, (UInt, ), sizeof(v)), since GraphBLAS will try to free using :jl_free not Libc.free.

2 Likes