Garbage collection is not working in this case on linux

Chong_Wang · February 4, 2018, 3:37am

I have a minimal working example here:

# This file is saved as ts.jl
struct NoTBModel
    norbits::Int64
    positions::Dict{Vector{Int64},Vector{Matrix{Float64}}}
end


function NoTBModel(norbits::Int64)
    return NoTBModel(
        norbits,
        Dict{Vector{Int64},Vector{Matrix{Float64}}}()
        )
end

function populateR(nm::NoTBModel, R::Vector{Int64})
    nm.positions[R] = [zeros(nm.norbits, nm.norbits) for i in 1:3]
    nm.positions[-R] = [zeros(nm.norbits, nm.norbits) for i in 1:3]
end

function createmodel()
    cellindex = [
        0   0   0;
       -1   0  -2;
       -1   0  -1;
       -1   0   0;
       -1   0   1;
       -1   0   2;
       -1   1  -1;
       -1   1   0;
        0  -1  -2;
        0  -1  -1;
    ]

    # cellindex = rand(Int64, 10, 3) # uncommenting this line solves the problem.

    nm = NoTBModel(
        2000,  # It seems changing 2000 to 2500 also solves the problem.
    )
    for i in 1:10
        populateR(nm, cellindex[i, :])
    end
    return nothing
end

Then in REPL, I type

julia> include("ts.jl")
createmodel (generic function with 1 method)

julia> createmodel()

julia> gc()

However, the memory usage of NoTBModel created in createmodel function is not released. Strangely, changing only one line of the code (see the comment in the code) solves the problem. I tracked the usage of memory through top in linux. This issue does NOT happen in Windows.

More info:

julia> versioninfo()
Julia Version 0.6.2
Commit d386e40* (2017-12-13 18:08 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
  WORD_SIZE: 64
  BLAS: libmkl_rt
  LAPACK: libmkl_rt
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, sandybridge)

Edit:
I can also reproduce this on

julia> versioninfo()
Julia Version 0.6.2
Commit d386e40 (2017-12-13 18:08 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas
  LIBM: libm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

jekyllstein · January 29, 2019, 8:31am

I have the same issue. I made a post to the issue tracker referencing your old post as well. Memory used by dictionary never freed on Linux · Issue #30888 · JuliaLang/julia · GitHub

I copied it here too:

This issue was experienced on Linux with the following version info and is related to issue #25884 filed by @mistguy :

Julia Version 1.0.3
Commit 1b5990863d (2019-01-15 20:28 UTC)
Platform Info:
OS: Linux (x86_64-unknown-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

When a dictionary is created in a function or global space that has a matrix as its value, its memory never gets freed for the Linux OS kernel to use. I tested this in and out of interactive mode with the same effect.
I created the following script to test on different systems and saved it as dictAllocationTest.jl to run it as shown below:

using InteractiveUtils
versioninfo()

println()
if isinteractive()
	println("Julia running in interactive mode")
else
	println("Julia not running in interactive mode")
end
println()
totMem = Int64(Sys.total_memory())

println()
if isempty(ARGS)
	println("Testing in global space")
else
	println("Testing inside a function")
end
println()

function createTestDict(totMem)
	testDict = Dict{Int64, Matrix{Float64}}()
	for i in 1:100000
		push!(testDict, i => ones(Float64, 100, 100))
	end
	mem2 = totMem - Int64(Sys.free_memory())
	testDict = ()
	return mem2
end

mem1 = totMem - Int64(Sys.free_memory())

if isempty(ARGS)
	testDict = Dict{Int64, Matrix{Float64}}()
	for i in 1:100000
		push!(testDict, i => ones(Float64, 100, 100))
	end
	mem2 = totMem - Int64(Sys.free_memory())
	testDict = ()
else
	mem2 = createTestDict(totMem)
end

GC.gc()

mem3 = totMem - Int64(Sys.free_memory())

println("Memory added from dictionary is $((mem2 - mem1)/(10^9)) gigabytes")
println("Memory freed from clearing dictionary is $((mem2 - mem3)/(10^9)) gigabytes")
println("Final memory usage is $(100*mem3/mem1) % of the original")

Here are the results of testing it on linux in the global space:

> $julia1 dictAllocationTest.jl
Julia Version 1.0.3
Commit 1b5990863d (2019-01-15 20:28 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia not running in interactive mode

Testing in global space
Memory added from dictionary is 8.016703488 gigabytes
Memory freed from clearing dictionary is 0.01390592 gigabytes
Final memory usage is 161.5230175061836 % of the original

and in the function space:

> $julia1 dictAllocationTest.jl jkl
Julia Version 1.0.3
Commit 1b5990863d (2019-01-15 20:28 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia not running in interactive mode

Testing inside a function
Memory added from dictionary is 8.023105536 gigabytes
Memory freed from clearing dictionary is 0.0 gigabytes
Final memory usage is 161.65566657769227 % of the original

In both cases memory is not freed after calling GC.gc() even after the function call is completed. In contrast when I test this on macOS the results are very different:

$ $julia1 dictAllocationTest.jl
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia not running in interactive mode


Testing in global space

Memory added from dictionary is 8.097398784 gigabytes
Memory freed from clearing dictionary is 7.736045568 gigabytes
Final memory usage is 100.62443512513228 % of the original

$ $julia1 dictAllocationTest.jl jkgf
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia not running in interactive mode


Testing inside a function

Memory added from dictionary is 8.09453568 gigabytes
Memory freed from clearing dictionary is 7.783051264 gigabytes
Final memory usage is 100.53822059089042 % of the original

On macOS and Windows the memory is freed in both cases. It does seem however that if GC.gc() is called inside the function then memory is not freed until it is called after the function completes as I do in the script. At present I don’t know how to free the memory without exiting the Julia session all together.

sdanisch · January 29, 2019, 10:00am

You need to call GC.gc(false) for a full swipe - otherwise it’s up to the heuristic to free anything!
with GC.gc(false), I get:

Memory added from dictionary is 8.027250688 gigabytes
Memory freed from clearing dictionary is 4.064083968 gigabytes
Final memory usage is 130.38160618678612 % of the original

There are still 4gb leftover, but it’s at least better

yuyichao · January 29, 2019, 10:46am

GC.gc(false) is not a full collection.

jekyllstein · January 29, 2019, 6:13pm

Interesting. That half freeing is also what I saw when I created a matrix M and explicitly placed a copy into the dictionary. I didn’t put it in the script to avoid confusion but I wonder why the performance differs so much.

bennedich · January 29, 2019, 8:12pm

What happens if you re-run the test repeatedly, does memory eventually get freed?

I’ve seen something similar with other objects (not dictionaries) – memory was not freed at all after GC, as defined by measuring memory usage of the Julia process – but if I kept allocating and freeing objects until the point where my system would run out of memory, it was all suddenly freed.

Palli · February 2, 2019, 8:39pm

EDIT: I misread, yes, GC.gc(true) is a full collection. Disregard the rest:

I believe you’re the go-to guy on the GC, and would know. I assume this recently changed, and what does it do now (is it simply the plan to get rid of this “feature”, just not dropped yet?)?

In my Julia 1.2 I see no parameter in the docs, while:

julia> @less GC.gc(false)  # shows (are comments in code outdated?):

gc(full::Bool=true) = ccall(:jl_gc_collect, Cvoid, (Int32,), full)

github.com

JuliaLang/julia/blob/24f1316e91de029f71f636db23aced49156b44ad/src/gc.c#L3292


      
              return jl_gc_alloc(ptls, sizeof(void*) * 3, NULL);
          }
          
          JL_DLLEXPORT int jl_gc_enable_conservative_gc_support(void)
          {
              static_assert(jl_buff_tag % GC_PAGE_SZ == 0,
                  "jl_buff_tag must be a multiple of GC_PAGE_SZ");
              if (jl_is_initialized()) {
                  int result = jl_atomic_fetch_or(&support_conservative_marking, 1);
                  if (!result) {
                      // Do a full collection to ensure that age bits are updated
                      // properly. We don't have to worry about race conditions
                      // for this part, as allocation itself is unproblematic and
                      // a collection will wait for safepoints.
                      jl_gc_collect(1);
                  }
                  return result;
              } else {
                  int result = support_conservative_marking;
                  support_conservative_marking = 1;
                  return result;

Off-topic, googling this all, I found rust crate for Julia:

jekyllstein · February 12, 2019, 10:37pm

I’ve tried running GC many times and clearing and putting new values into the object but the memory isn’t freed. In this case the linux system did run out of memory as well since it lacks any swap so GC wasn’t triggered when memory was going to run out either.