Ever increasing time for garbage collection

ufechner7 · May 23, 2022, 9:48pm

Hello,
I have a simulation that is running in real-time for an undefined period of time (until you stop it).
I solve my equation system every 50ms. For one solution about 800 calls to the inner residual function are needed. This works fine.

The problem I have is that the time needed for the garbage collection is continuously increasing.
joystick

I do an incremental garbage collection once per time step like this:

t_gc_tot = @elapsed GC.gc(false)

After 300s of simulation time already 12ms time are needed for the garbage collection, and if you extrapolate after 1200s the simulation will fail because most of the time is used for gc.

What can I do to solve this issue?

I have zero allocations in my simulation code, but the solver (IDA solver from Sundials with GMRES linear solver) is allocating a lot.

Oscar_Smith · May 23, 2022, 9:53pm

That’s really not supposed to happen. If you get a mwe that reproduces the behavior, it would be good to have it in GitHub - JuliaCI/GCBenchmarks

ufechner7 · May 23, 2022, 10:04pm

If you check out this repository: GitHub - aenarete/KiteViewers.jl: 3D viewer for airborne wind energy systems
And then execute:

include("examples/depower_simple.jl")

you get a plot where you can already see the increasing gc time…

But without active control you cannot run the simulation longer than 50s. If you have a joystick you can control the kite and run:

include("examples/joystick.jl")

which I did to produce the example in the last post.

But for a nice MWE I would need an autopilot which might be ready in a week…

vchuravy · May 23, 2022, 10:08pm

It would be important to understand if the amount of memory stays constant or also increases evermore. The GC should only increase as function of heap_size and object-graph size. So this sounds like something is allocating more and more, but never releasing any of that memory,

ufechner7 · May 23, 2022, 10:14pm

I can try to disable the Makie based GUI and see if the problem persists…

How can I measure the memory that is used from within Julia?

Palli · May 23, 2022, 10:22pm

Did you maybe hit this:

https://github.com/JuliaLang/julia/issues/45068

ufechner7 · May 23, 2022, 10:32pm

No, I did not turn GC off in my test…

Palli · May 23, 2022, 11:10pm

Then I would look into is it also deallocating a lot (and look at max. mem. used, somehow).

I.e. I would look into if you’re using it wrong, or even if it has a bug. I suppose the underlying C code is ok.

I did look up e.g. Alloc, free, malloc (only found some constants that way), in the wrapper, and found some matches for all.

Do you need to do something with handles?

github.com

SciML/Sundials.jl/blob/b88762bdaf4141d1c4363cff8f6175cfca937f39/src/handle.jl#L48


      
          
             Wraps the reference to the pointer to the Sundials object.
             Manages automatic destruction of the referenced objects when it is
             no longer in use.
          """
          struct Handle{T <: AbstractSundialsObject} <: SundialsHandle
              ptr_ref::Ref{Ptr{T}} # pointer to a pointer
          
              function Handle(ptr::Ptr{T}) where {T<:AbstractSundialsObject}
                  h = new{T}(Ref{Ptr{T}}(ptr))
                  finalizer(release_handle, h.ptr_ref)
                  return h
              end
          end
          
          mutable struct MatrixHandle{T <: SundialsMatrix} <: SundialsHandle
              ptr::SUNMatrix
              destroyed::Bool
              function MatrixHandle(ptr::SUNMatrix, M::T) where {T<:SundialsMatrix}
                  h = new{T}(ptr, false)
                  finalizer(release_handle, h)

I’m not sure what this default early_free = false is, but likely a false alarm:

github.com

SciML/Sundials.jl/blob/b88762bdaf4141d1c4363cff8f6175cfca937f39/src/common_interface/solve.jl#L1467


      
          function get_iters!(integrator::CVODEIntegrator, iters)
              CVodeGetNumSteps(integrator.mem, iters)
          end
          function get_iters!(integrator::ARKODEIntegrator, iters)
              ARKStepGetNumSteps(integrator.mem, iters)
          end
          function get_iters!(integrator::IDAIntegrator, iters)
              IDAGetNumSteps(integrator.mem, iters)
          end
          
          function DiffEqBase.solve!(integrator::AbstractSundialsIntegrator; early_free = false,
                                     calculate_error = false)
              uType = eltype(integrator.sol.u)
              iters = Ref(Clong(-1))
              while !isempty(integrator.opts.tstops)
                  # Sundials can have floating point issues approaching a tstop if
                  # there is a modifying event each
                  # The call to first is an overload of Base.first implemented in DataStructures
                  while integrator.tdir * (integrator.t - first(integrator.opts.tstops)) < -1e6eps()
                      tstop = first(integrator.opts.tstops)
                      set_stop_time(integrator, tstop)