Memory usage discrepancy

General Usage
,
1

I have an application running in Julia.

Using htop I can see that the process consumes 2 GB of memory, but if I use summarysize of all symbols in Main I just get around 150 MB in total. How can this be?

I’m using Julia v1.11.2

2

I have also found summarysize to be of little use trying to find out where memory is stashed. I was once pointed to this snippet by @gbaraldi

function take_heap_snapshot(filename::String)
    flags = Base.open_flags(
        read = true,
        write = true,
        create = true,
        truncate = true,
        append = false,
    )
    nodes = IOStream("<file $filename.nodes>")
    ccall(:ios_file, Ptr{Cvoid}, (Ptr{UInt8}, Cstring, Cint, Cint, Cint, Cint),
        nodes.ios, "$filename.nodes", flags.read, flags.write, flags.create, flags.truncate)
    edges = IOStream("<file $filename.edges>")
    ccall(:ios_file, Ptr{Cvoid}, (Ptr{UInt8}, Cstring, Cint, Cint, Cint, Cint),
        edges.ios, "$filename.edges", flags.read, flags.write, flags.create, flags.truncate)
    strings = IOStream("<file $filename.strings>")
    ccall(:ios_file, Ptr{Cvoid},(Ptr{UInt8}, Cstring, Cint, Cint, Cint, Cint),
        strings.ios, "$filename.strings", flags.read, flags.write, flags.create, flags.truncate)
    json = IOStream("<file $filename.metadata.json>")
    ccall(:ios_file, Ptr{Cvoid}, (Ptr{UInt8}, Cstring, Cint, Cint, Cint, Cint),
        json.ios, "$filename.metadata.json", flags.read, flags.write, flags.create, flags.truncate)
    ccall(:jl_gc_take_heap_snapshot,
        Cvoid,
        (Ptr{Cvoid},Ptr{Cvoid},Ptr{Cvoid},Ptr{Cvoid}, Cchar),
        nodes.handle, edges.handle, strings.handle, json.handle,
        Cchar(false))
    ccall(:ios_close, Cint, (Ptr{Cvoid},), nodes.ios)
    ccall(:ios_close, Cint, (Ptr{Cvoid},), edges.ios)
    ccall(:ios_close, Cint, (Ptr{Cvoid},), strings.ios)
    ccall(:ios_close, Cint, (Ptr{Cvoid},), json.ios)
    return nothing
end

if you call this function somewhere and then call

Profile.Heapsnapshot.assemble

with the first argument to assemble is the prefix you passed in to take_heap_snapshot, and the second is <filename>.heapsnapshot, like this

 Profile.HeapSnapshot.assemble_snapshot("heap_snapshot", "heap_snapshot.heapsnapshot")

You should be able to look at the heap snapshot in vscode or in chrome by pressing F12, go to memory and heap snapshot

Thanks to @gbaraldi for those instructions, I mostly copied them from him, and he might be able to provide further insights if you get stuck or have quesitons I cannot answer.

1 Like
3

  1. Names in Main don’t account for every symbol in a Julia process, there are other modules.

  2. There are many relevant things to an object that Base.summarysize doesn’t count; note that the docstring for summarysize says “all unique objects reachable”. Think of a function: it has a type, multiple methods, each method can have several compiled versions, the compiler itself must also take up memory. But those are known to be shared by every instance of the function, so summarysize will be much lower, often 0.

  3. I’m not sure if you’re doing this, but adding up results of summarysize called for each symbol separately would overcount reached unique objects. Given x = collect(1:1000); y = view(x, :), we know most of the memory is contributed by x, and y adds very little. However, Base.summarysize(x) + Base.summarysize(y) counts x’s memory twice, whereas Base.summarysize((x,y)) does not.

3 Likes
4

You needn’t use this version of this code. This was just for --trim.
Profile.take_heap_snapshot does the same thing but easier/nicer

3 Likes
5

For htop are you looking at virtual or resident memory? Julia is fairly careless with virtual memory since it tends to be almost completely free (since unix systems won’t actually give you the pages until you ask for them).

1 Like
6

I took a heap snapshot. It is 620 MB large. But to be honest, I cannot do much with it.

image
image935×220 14.8 KB

If I start Julia and activate my environment, it is 230 MB.

If I compare the two I see this:

image
image980×362 26.5 KB

Is (String) my Code that is loaded?

7

I definitely have a memory leak somewhere. This is a Grafana screenshot. I added logging of my summarysize approach (which is entirely wrong) and what I get in total RSS with ps.

image
image460×304 10 KB

My app uses HiGHS.jl, which might be the source. Since JuMP.jl creates the optimizer on the fly, it needs to be destroyed as well: HiGHS.jl/src/MOI_wrapper.jl at 577ecc26040fb2a8186cd51f4850c8c41c21c98a · jump-dev/HiGHS.jl · GitHub

1 Like
8

Letting it run overnight I see that the memory usage flattens out:

image
image482×302 10.1 KB

I would guess that there is some problem with the GC. I will try with --heap-size-hint.

9

Running with --heap-size-hint=1G and a GC.gc() after the optimization takes place solved the issue. It is now a steady 1.7 GB.