Why is garbage collection called here?

General Usage Performance
1

Hello!

I’ve been hitting my head on the wall for a few days now. I have a function which when using @profview seems to call the garbage collection (red bar), but it does not allocate. It is updatexij! (which is resized every iteration in ResizeBuffers!) in the following screenshot:

image
image890Γ—342 23.9 KB

When I look into the function it self:

image
image793Γ—319 22.9 KB

No red bars are present.

How do I debug this and figure out where the garbage collection is coming from?

Kind regards

2

I don’t think that garbage collection is generally attributed to the lines that did the allocation because GC runs at essentially random times.

Other points to check:

  • is everything type stable at the call site? Dynamic dispatch causes allocations
  • you have wrapped things in @timeit that might cause allocation maybe? Do you still see the GC runs if you remove the timing macros?
  • are the GC timings/valuea the same between runs?
  • GC generally runs when there are allocations. So to reduce GC you need to reduce allocations. You can also track allocations. See here
    Profiling Β· The Julia Language
1 Like
3

Okay got it, it is just because these are pretty much the only places I see GC in my code.

  • is everything type stable at the call site? Dynamic dispatch causes allocations

I believe so. Using JET.

julia> @report_opt updatexα΅’β±Ό!(xα΅’β±Ό,system.nb.list,Position)
No errors detected


julia> @report_call updatexα΅’β±Ό!(xα΅’β±Ό,system.nb.list,Position)
No errors detected

Using @code_warntype:

@code_warntype updatexα΅’β±Ό!(xα΅’β±Ό,system.nb.list,Position)
MethodInstance for SPHExample.SimulationEquations.updatexα΅’β±Ό!(::Vector{SVector{3, Float64}}, ::Vector{Tuple{Int64, Int64, Float64}}, ::Vector{SVector{3, Float64}})
  from updatexα΅’β±Ό!(xα΅’β±Ό, list, points) @ SPHExample.SimulationEquations c:\git\SPHExample\src\SimulationEquations.jl:220
Arguments
  #self#::Core.Const(SPHExample.SimulationEquations.updatexα΅’β±Ό!)
  xα΅’β±Ό::Vector{SVector{3, Float64}}
  list::Vector{Tuple{Int64, Int64, Float64}}
  points::Vector{SVector{3, Float64}}
Locals
  @_5::Union{Nothing, Tuple{Tuple{Int64, Tuple{Int64, Int64, Float64}}, Tuple{Int64, Int64}}}
  @_6::Int64
  L::Tuple{Int64, Int64, Float64}
  iter::Int64
  j::Int64
  i::Int64
Body::Nothing
1 ─       Core.NewvarNode(:(@_5))
β”‚   %2  = SPHExample.SimulationEquations.length(xα΅’β±Ό)::Int64
β”‚   %3  = SPHExample.SimulationEquations.length(list)::Int64
β”‚   %4  = (%2 != %3)::Bool
└──       goto #3 if not %4
2 ─ %6  = SPHExample.SimulationEquations.length(list)::Int64
└──       SPHExample.SimulationEquations.resize!(xα΅’β±Ό, %6)
3 β”„ %8  = SPHExample.SimulationEquations.enumerate(list)::Base.Iterators.Enumerate{Vector{Tuple{Int64, Int64, Float64}}}
β”‚         (@_5 = Base.iterate(%8))
β”‚   %10 = (@_5 === nothing)::Bool
β”‚   %11 = Base.not_int(%10)::Bool
└──       goto #6 if not %11
4 β”„ %13 = @_5::Tuple{Tuple{Int64, Tuple{Int64, Int64, Float64}}, Tuple{Int64, Int64}}
β”‚   %14 = Core.getfield(%13, 1)::Tuple{Int64, Tuple{Int64, Int64, Float64}}
β”‚   %15 = Base.indexed_iterate(%14, 1)::Core.PartialStruct(Tuple{Int64, Int64}, Any[Int64, Core.Const(2)])
β”‚         (iter = Core.getfield(%15, 1))
β”‚         (@_6 = Core.getfield(%15, 2))
β”‚   %18 = Base.indexed_iterate(%14, 2, @_6::Core.Const(2))::Core.PartialStruct(Tuple{Tuple{Int64, Int64, Float64}, Int64}, Any[Tuple{Int64, Int64, Float64}, Core.Const(3)])
β”‚         (L = Core.getfield(%18, 1))
β”‚   %20 = Core.getfield(%13, 2)::Tuple{Int64, Int64}
β”‚         (i = Base.getindex(L, 1))
β”‚         (j = Base.getindex(L, 2))
β”‚   %23 = Base.getindex(points, i)::SVector{3, Float64}
β”‚   %24 = Base.getindex(points, j)::SVector{3, Float64}
β”‚   %25 = (%23 - %24)::SVector{3, Float64}
β”‚         Base.setindex!(xα΅’β±Ό, %25, iter)
β”‚         (@_5 = Base.iterate(%8, %20))
β”‚   %28 = (@_5 === nothing)::Bool
β”‚   %29 = Base.not_int(%28)::Bool
└──       goto #6 if not %29
5 ─       goto #4
6 β”„       return nothing
  • you have wrapped things in @timeit that might cause allocation maybe? Do you still see the GC runs if you remove the timing macros?

Commented out @timeit, still see GC:

image
image932Γ—185 15.6 KB

  • are the GC timings/valuea the same between runs?

Yes, similar.

  • GC generally runs when there are allocations. So to reduce GC you need to reduce allocations. You can also track allocations. See here

I’ve done @benchmark on the function and it has zero allocations. This is why I struggle to find out the root cause.

I would really like to fix this for max performance, so any help appreciated :slight_smile:

4

That is of course also good information but not what I meant. Sorry for being unclear. Dynamic dispatch would occur when you call updatexα΅’β±Ό if the function surrounding the call has type instabilities. So suppose you have the code snippets you showed inside some larger function:

function large_function(...)
    #...
    updatexα΅’β±Ό!(xα΅’β±Ό,system.nb.list,Position)
    # ... more code
end

You showed that updatexα΅’β±Ό! is good (does not allocate and is type stable). So now we need to check the large_function for allocations/type instabilities :slight_smile:

2 Likes
5

Got it, thanks!

I now do it on the large_function in my case called RunSimulation,

@report_opt target_modules=(@__MODULE__,):

═════ 18 possible errors found ═════
β”Œ kwcall(::@NamedTuple{FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}}, ::typeof(RunSimulation)) @ Main c:\git\SPHExample\example\MainSimulation.jl:58
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:74 
β”‚β”‚ runtime dispatch detected: Core.kwcall(%16::@NamedTuple{NumberOfParticles::Int64}, %14::Type{SimulationDataResults{3, _A}} where _A)::SimulationDataResults{3}
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:78 
β”‚β”‚ runtime dispatch detected: vcat(%53::AbstractVector, %54::AbstractVector)::Any
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:78 
β”‚β”‚ runtime dispatch detected: Array(%55::Any)::Any
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:78 
β”‚β”‚ runtime dispatch detected: Base.broadcasted(identity, %56::Any)::Any
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:78 
β”‚β”‚ runtime dispatch detected: Base.materialize!(%20::Vector, %57::Any)::Any
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:95 
β”‚β”‚ runtime dispatch detected: create_vtp_file(SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}, %17::SimulationDataResults{3})::Any
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:102β”‚β”‚ runtime dispatch detected: zeros(%3::DataType, %718::Tuple{Int64})::Vector
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:103β”‚β”‚ runtime dispatch detected: zeros(%734::Union{Type{Any}, Type{E} where E<:(SVector{3})}, %720::Tuple{Int64})::Vector{T} where T<:(SVector{3})
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:105β”‚β”‚ runtime dispatch detected: zeros(%3::DataType, %718::Tuple{Int64})::Vector
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:106β”‚β”‚ runtime dispatch detected: zeros(%734::Union{Type{Any}, Type{E} where E<:(SVector{3})}, %720::Tuple{Int64})::Vector{T} where T<:(SVector{3})
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:107β”‚β”‚ runtime dispatch detected: zeros(%734::Union{Type{Any}, Type{E} where E<:(SVector{3})}, %720::Tuple{Int64})::Vector{T} where T<:(SVector{3})
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:109β”‚β”‚ runtime dispatch detected: zeros(%3::DataType, %718::Tuple{Int64})::Vector
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:112β”‚β”‚ runtime dispatch detected: zeros(%734::Union{Type{Any}, Type{E} where E<:(SVector{3})}, %720::Tuple{Int64})::Vector{T} where T<:(SVector{3})
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:113β”‚β”‚ runtime dispatch detected: zeros(%734::Union{Type{Any}, Type{E} where E<:(SVector{3})}, %720::Tuple{Int64})::Vector{T} where T<:(SVector{3})
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:114β”‚β”‚ runtime dispatch detected: zeros(%3::DataType, %718::Tuple{Int64})::Vector
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:115β”‚β”‚ runtime dispatch detected: zeros(%3::DataType, %718::Tuple{Int64})::Vector
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:117β”‚β”‚ runtime dispatch detected: zeros(%3::DataType, %718::Tuple{Int64})::Vector
│└────────────────────
β”‚β”Œ RunSimulation(; FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}) @ Main c:\git\SPHExample\example\MainSimulation.jl:120β”‚β”‚ runtime dispatch detected: Core.kwcall(%791::NamedTuple{(:x, :cutoff, :parallel), <:Tuple{Array{SVector{3, _A}, 1} where _A, Float64, Bool}}, InPlaceNeighborList)::InPlaceNeighborList{B, _A, _B, CellListMap.NeighborList{Float64}} where {B<:(Box{NonPeriodicCell}), _A, _B}
│└────────────────────

@report_call target_modules=(@__MODULE__,):
No errors detected

@code_warntype:

MethodInstance for Core.kwcall(::@NamedTuple{FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}}, ::typeof(RunSimulation))
  from kwcall(::NamedTuple, ::typeof(RunSimulation)) @ Main c:\git\SPHExample\example\MainSimulation.jl:58
Arguments
  _::Core.Const(Core.kwcall)
  @_2::@NamedTuple{FluidCSV::String, BoundCSV::String, SimulationMetaData::SimulationMetaData, SimulationConstants::SimulationConstants{Float64}}
  @_3::Core.Const(RunSimulation)
Locals
  @_4::Union{SimulationConstants{Float64}, String, SimulationMetaData}
  FluidCSV::String
  BoundCSV::String
  SimulationMetaData::SimulationMetaData
  SimulationConstants::SimulationConstants{Float64}
Body::Bool
1 ──       Core.NewvarNode(:(@_4))
β”‚    %2  = Core.isdefined(@_2, :FluidCSV)::Core.Const(true)
└───       goto #6 if not %2
2 ── %4  = Core.getfield(@_2, :FluidCSV)::String
β”‚    %5  = (%4 isa Main.String)::Core.Const(true)
└───       goto #4 if not %5
3 ──       goto #5
4 ──       Core.Const(:(%new(Core.TypeError, Symbol("keyword argument"), :FluidCSV, Main.String, %4)))
└───       Core.Const(:(Core.throw(%8)))
5 ┄─       (@_4 = %4)
└───       goto #7
6 ──       Core.Const(:(Core.UndefKeywordError(:FluidCSV)))
└───       Core.Const(:(@_4 = Core.throw(%12)))
7 ┄─ %14 = @_4::String
β”‚          (FluidCSV = %14)
β”‚    %16 = Core.isdefined(@_2, :BoundCSV)::Core.Const(true)
└───       goto #12 if not %16
8 ── %18 = Core.getfield(@_2, :BoundCSV)::String
β”‚    %19 = (%18 isa Main.String)::Core.Const(true)
└───       goto #10 if not %19
9 ──       goto #11
10 ─       Core.Const(:(%new(Core.TypeError, Symbol("keyword argument"), :BoundCSV, Main.String, %18)))
└───       Core.Const(:(Core.throw(%22)))
11 β”„       (@_4 = %18)
└───       goto #13
12 ─       Core.Const(:(Core.UndefKeywordError(:BoundCSV)))
└───       Core.Const(:(@_4 = Core.throw(%26)))
13 β”„ %28 = @_4::String
β”‚          (BoundCSV = %28)
β”‚    %30 = Core.isdefined(@_2, :SimulationMetaData)::Core.Const(true)
└───       goto #18 if not %30
14 ─ %32 = Core.getfield(@_2, :SimulationMetaData)::SimulationMetaData
β”‚    %33 = (%32 isa Main.SimulationMetaData)::Core.Const(true)
└───       goto #16 if not %33
15 ─       goto #17
16 ─       Core.Const(:(%new(Core.TypeError, Symbol("keyword argument"), :SimulationMetaData, Main.SimulationMetaData, %32)))
└───       Core.Const(:(Core.throw(%36)))
17 β”„       (@_4 = %32)
└───       goto #19
18 ─       Core.Const(:(Core.UndefKeywordError(:SimulationMetaData)))
└───       Core.Const(:(@_4 = Core.throw(%40)))
19 β”„ %42 = @_4::SimulationMetaData
β”‚          (SimulationMetaData = %42)
β”‚    %44 = Core.isdefined(@_2, :SimulationConstants)::Core.Const(true)
└───       goto #24 if not %44
20 ─ %46 = Core.getfield(@_2, :SimulationConstants)::SimulationConstants{Float64}
β”‚    %47 = (%46 isa Main.SimulationConstants)::Core.Const(true)
└───       goto #22 if not %47
21 ─       goto #23
22 ─       Core.Const(:(%new(Core.TypeError, Symbol("keyword argument"), :SimulationConstants, Main.SimulationConstants, %46)))
└───       Core.Const(:(Core.throw(%50)))
23 β”„       (@_4 = %46)
└───       goto #25
24 ─       Core.Const(:(Core.UndefKeywordError(:SimulationConstants)))
└───       Core.Const(:(@_4 = Core.throw(%54)))
25 β”„ %56 = @_4::SimulationConstants{Float64}
β”‚          (SimulationConstants = %56)
β”‚    %58 = (:FluidCSV, :BoundCSV, :SimulationMetaData, :SimulationConstants)::Core.Const((:FluidCSV, :BoundCSV, :SimulationMetaData, :SimulationConstants))
β”‚    %59 = Core.apply_type(Core.NamedTuple, %58)::Core.Const(NamedTuple{(:FluidCSV, :BoundCSV, :SimulationMetaData, :SimulationConstants)})
β”‚    %60 = Base.structdiff(@_2, %59)::Core.Const(NamedTuple())
β”‚    %61 = Base.pairs(%60)::Core.Const(Base.Pairs{Symbol, Union{}, Tuple{}, @NamedTuple{}}())
β”‚    %62 = Base.isempty(%61)::Core.Const(true)
└───       goto #27 if not %62
26 ─       goto #28
27 ─       Core.Const(:(Base.kwerr(@_2, @_3)))
28 β”„ %66 = Main.:(var"#RunSimulation#25")(FluidCSV, BoundCSV, SimulationMetaData, SimulationConstants, @_3)::Core.Const(false)
└───       return %66

Is this more a long the lines what you asked for? :slight_smile:

6

Yes it is :slight_smile:
And we see that there is indeed some type-instability here. Not sure whether it is relevant just yet. The @code_warntype actually just shows the types inside a β€œwrapper” that Julia generated to handle keyword and default arguments. I am unsure how the Jet.jl output needs to be interpreted, i.e. whether that means there truly is type instability inside RunSimulation or just in the β€œwrapper” function. So we need to get the @code_warntype of the inner function. You can do this with Cthulhu.jl. This allows you to recursively descend into the function calls and print the code_warntype. So instead of @code_warntype RunSimulation(...) you use @descend RunSimulation(...) and then you can interactively descend into the call.
Disclaimer: I have so far not done this myself.

Can I see the full code of RunSimulation somewhere or can you post it? That would help me to get a feeling for what’s happening :slight_smile:

7

Yes!

The code for RunSimulation is here: SPHExample/example/MainSimulation.jl at optimize-code-execution Β· AhmedSalih3d/SPHExample Β· GitHub

The first part of the function loads in some stuff and allocates once, then the main loop inside of the function calculates different properties.

Kind regards

8

Ok that makes everything clearer. RunSimulation is infact quite type unstable :slight_smile:
See you pass in SimulationMetaData and that contains abstractly typed fields (CurrentTimeStep::AbstractFloat, TotalTime::AbstractFloat) and even types in variables (FloatType::DataType = Float64, IntType::DataType = Int64) that Julia cannot know at compile time of RunSimulation. I didn’t check the other arguments to RunSimulation.

There are a couple of ways out from here:

  • You could move all relevant type information actually into the types of the arguments to RunSimulation, i.e. make SimulationMetaData a parametric type like so:
@with_kw mutable struct SimulationMetaData{F, FloatType, IntType} where {FloatType <: AbstractFloat, IntType <: Integer}
    SimulationName::String
    SaveLocation::String
    HourGlass::TimerOutput           = TimerOutput()
    Iteration::Int                   = 0
    MaxIterations::Int               = 1000
    OutputIteration::Int             = 50
    CurrentTimeStep::F   = 0.0
    TotalTime::F         = 0.0
    SilentOutput::Bool               = false
    ThreadsCPU::Int                  = Threads.nthreads()
    ProgressSpecification::Progress  = Progress(MaxIterations)       
end

and do the same for the other parameters.

  • Another idea would be to use a function barrier. That means you restructure your code a bit and preallocate all things inside RunSimulation and then move the actual computation to a β€œkernel function” (called e.g. β€œRunSimulation_kernel”) which is called by RunSimulation with a ridiculous amount of arguments :smiley: Like so
function RunSimulation(;FluidCSV::String,
                        BoundCSV::String,
                        SimulationMetaData::SimulationMetaData,
                        SimulationConstants::SimulationConstants
)
    @unpack ...
    # do all the preallocation stuff
    # ....
    Pressureα΅’         = zeros(FloatType,         SizeOfParticlesI1)

    # Initialize the system system.nb.list
    system  = InPlaceNeighborList(x=Position, cutoff=2*h, parallel=true)
    
    # now call the kernel function to perform the actual computation
    # this causes a single dynamic dispatch and after that everything is type stable
    RunSimulation_kernel(system, Pressureα΅’, ... ,... )

    # now the rest of RunSimulation
    show(HourGlass,sortby=:name)
    show(HourGlass)
    disable_timer!(HourGlass)
end

function RunSimulation_kernel(....)
    @inbounds for SimulationMetaData.Iteration = 1:MaxIterations
    # Be sure to update and retrieve the updated neighbour system.nb.list at each time step
   # ... rest of the loops and computation
end

The latter has the great advantage that you don’t need to change a lot of code to make the computation fast. Although the setup/preallocation will stay type unstable but that doesn’t really matter as long as the computation takes the majority of the runtime anyways.

4 Likes
9

What I don’t like about the second suggestion is that it has to have these 100s of arguments… so I went ahead and gave your first suggestion a shot, since that was quite easy to do. Alhamdu lillah I did that! I paramterized SimulationMetaData similar to what you suggested:

@with_kw mutable struct SimulationMetaData{FloatType <: AbstractFloat}
    SimulationName::String
    SaveLocation::String
    HourGlass::TimerOutput           = TimerOutput()
    Iteration::Int                   = 0
    MaxIterations::Int               = 1000
    OutputIteration::Int             = 50
    CurrentTimeStep::FloatType       = 0
    TotalTime::FloatType             = 0
    SilentOutput::Bool               = false
    ThreadsCPU::Int                  = Threads.nthreads()
    ProgressSpecification::Progress  = Progress(MaxIterations)       
end

And understood what you meant about the compiler not knowing the datatype when it is a field instead of a type parameter

Then I went ahead and did the checks:

@report_opt target_modules=(@__MODULE__,)

image
image1584Γ—215 43.3 KB

Which is basically this line:

Density .= Array([DF_FLUID.Rhop;DF_BOUND.Rhop])

Any suggestion on how to fix?

@report_call target_modules=(@__MODULE__,)
No errors detected

@code_warntype

MethodInstance for Core.kwcall(::@NamedTuple{FluidCSV::String, BoundCSV::String, SimMetaData::SimulationMetaData{Float64}, SimConstants::SimulationConstants{Float64}}, ::typeof(RunSimulation))
  from kwcall(::NamedTuple, ::typeof(RunSimulation)) @ Main c:\git\SPHExample\example\MainSimulation.jl:58
Arguments
  _::Core.Const(Core.kwcall)
  @_2::@NamedTuple{FluidCSV::String, BoundCSV::String, SimMetaData::SimulationMetaData{Float64}, SimConstants::SimulationConstants{Float64}}
  @_3::Core.Const(RunSimulation)
Locals
  @_4::Union{SimulationConstants{Float64}, String, SimulationMetaData{Float64}}
  FluidCSV::String
  BoundCSV::String
  SimMetaData::SimulationMetaData{Float64}
  SimConstants::SimulationConstants{Float64}
Body::Bool
1 ──       Core.NewvarNode(:(@_4))
β”‚    %2  = Core.isdefined(@_2, :FluidCSV)::Core.Const(true)
└───       goto #6 if not %2
2 ── %4  = Core.getfield(@_2, :FluidCSV)::String
β”‚    %5  = (%4 isa Main.String)::Core.Const(true)
└───       goto #4 if not %5
3 ──       goto #5
4 ──       Core.Const(:(%new(Core.TypeError, Symbol("keyword argument"), :FluidCSV, Main.String, %4)))
└───       Core.Const(:(Core.throw(%8)))
5 ┄─       (@_4 = %4)
└───       goto #7
6 ──       Core.Const(:(Core.UndefKeywordError(:FluidCSV)))
└───       Core.Const(:(@_4 = Core.throw(%12)))
7 ┄─ %14 = @_4::String
β”‚          (FluidCSV = %14)
β”‚    %16 = Core.isdefined(@_2, :BoundCSV)::Core.Const(true)
└───       goto #12 if not %16
8 ── %18 = Core.getfield(@_2, :BoundCSV)::String
β”‚    %19 = (%18 isa Main.String)::Core.Const(true)
└───       goto #10 if not %19
9 ──       goto #11
10 ─       Core.Const(:(%new(Core.TypeError, Symbol("keyword argument"), :BoundCSV, Main.String, %18)))
└───       Core.Const(:(Core.throw(%22)))
11 β”„       (@_4 = %18)
└───       goto #13
12 ─       Core.Const(:(Core.UndefKeywordError(:BoundCSV)))
└───       Core.Const(:(@_4 = Core.throw(%26)))
13 β”„ %28 = @_4::String
β”‚          (BoundCSV = %28)
β”‚    %30 = Core.isdefined(@_2, :SimMetaData)::Core.Const(true)
└───       goto #18 if not %30
14 ─ %32 = Core.getfield(@_2, :SimMetaData)::SimulationMetaData{Float64}
β”‚    %33 = (%32 isa Main.SimulationMetaData)::Core.Const(true)
└───       goto #16 if not %33
15 ─       goto #17
16 ─       Core.Const(:(%new(Core.TypeError, Symbol("keyword argument"), :SimMetaData, Main.SimulationMetaData, %32)))
└───       Core.Const(:(Core.throw(%36)))
17 β”„       (@_4 = %32)
└───       goto #19
18 ─       Core.Const(:(Core.UndefKeywordError(:SimMetaData)))
└───       Core.Const(:(@_4 = Core.throw(%40)))
19 β”„ %42 = @_4::SimulationMetaData{Float64}
β”‚          (SimMetaData = %42)
β”‚    %44 = Core.isdefined(@_2, :SimConstants)::Core.Const(true)
└───       goto #24 if not %44
20 ─ %46 = Core.getfield(@_2, :SimConstants)::SimulationConstants{Float64}
β”‚    %47 = (%46 isa Main.SimulationConstants)::Core.Const(true)
└───       goto #22 if not %47
21 ─       goto #23
22 ─       Core.Const(:(%new(Core.TypeError, Symbol("keyword argument"), :SimConstants, Main.SimulationConstants, %46)))
└───       Core.Const(:(Core.throw(%50)))
23 β”„       (@_4 = %46)
└───       goto #25
24 ─       Core.Const(:(Core.UndefKeywordError(:SimConstants)))
└───       Core.Const(:(@_4 = Core.throw(%54)))
25 β”„ %56 = @_4::SimulationConstants{Float64}
β”‚          (SimConstants = %56)
β”‚    %58 = (:FluidCSV, :BoundCSV, :SimMetaData, :SimConstants)::Core.Const((:FluidCSV, :BoundCSV, :SimMetaData, :SimConstants))
β”‚    %59 = Core.apply_type(Core.NamedTuple, %58)::Core.Const(NamedTuple{(:FluidCSV, :BoundCSV, :SimMetaData, :SimConstants)})
β”‚    %60 = Base.structdiff(@_2, %59)::Core.Const(NamedTuple())
β”‚    %61 = Base.pairs(%60)::Core.Const(Base.Pairs{Symbol, Union{}, Tuple{}, @NamedTuple{}}())
β”‚    %62 = Base.isempty(%61)::Core.Const(true)
└───       goto #27 if not %62
26 ─       goto #28
27 ─       Core.Const(:(Base.kwerr(@_2, @_3)))
28 β”„ %66 = Main.:(var"#RunSimulation#101")(FluidCSV, BoundCSV, SimMetaData, SimConstants, @_3)::Core.Const(false)
└───       return %66

And finally for the big reveal!

No GC happening!

image
image835Γ—146 25.7 KB

And now look at my @timeroutput measurements, all allocations pretty much vanished and the ones left I know why!

                                                                      Time                    Allocations
                                                             ───────────────────────   ────────────────────────
                      Tot / % measured:                            162s /  91.6%           1.19GiB /  38.4%

 Section                                             ncalls     time    %tot     avg     alloc    %tot      avg
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
 2| DDT                                               10.0k    29.2s   19.7%  2.92ms   27.1MiB    5.8%  2.78KiB
 2| DDT2                                              10.0k    28.6s   19.3%  2.86ms   26.4MiB    5.6%  2.70KiB
 0 | Update Neighbour system.nb.list                  10.0k    22.5s   15.2%  2.25ms    164MiB   35.1%  16.8KiB
 1 | Update xα΅’β±Ό, kernel values and kernel gradient    10.0k    20.7s   14.0%  2.07ms     0.00B    0.0%    0.00B
 4| OutputVTP                                         10.0k    11.0s    7.4%  1.10ms    243MiB   52.0%  24.9KiB
 2| βˆ‚Ξ α΅’β±Όβˆ‚t!                                           10.0k    8.97s    6.0%   896ΞΌs     0.00B    0.0%    0.00B
 2| βˆ‚Ξ α΅’β±Όβˆ‚t!2                                          10.0k    8.95s    6.0%   895ΞΌs     0.00B    0.0%    0.00B
 2| βˆ‚vα΅’βˆ‚t!                                            10.0k    5.17s    3.5%   517ΞΌs     0.00B    0.0%    0.00B
 2| βˆ‚vα΅’βˆ‚t!2                                           10.0k    5.17s    3.5%   517ΞΌs     0.00B    0.0%    0.00B
 2| updatexα΅’β±Ό!                                        10.0k    4.04s    2.7%   404ΞΌs     0.00B    0.0%    0.00B
 0 | Reset arrays to zero and resize L arrays         10.0k    699ms    0.5%  69.8ΞΌs   7.03MiB    1.5%     737B
 2| Pressure                                          10.0k    546ms    0.4%  54.6ΞΌs     0.00B    0.0%    0.00B
 2| Pressure2                                         10.0k    544ms    0.4%  54.4ΞΌs     0.00B    0.0%    0.00B
 3| Calculating time step                             10.0k    520ms    0.4%  52.0ΞΌs     0.00B    0.0%    0.00B
 2| Positionₙ⁺                                        10.0k    345ms    0.2%  34.5ΞΌs     0.00B    0.0%    0.00B
 2| Position                                          10.0k    306ms    0.2%  30.6ΞΌs     0.00B    0.0%    0.00B
 2| vₙ⁺                                               10.0k    196ms    0.1%  19.6ΞΌs     0.00B    0.0%    0.00B
 2| Velocity                                          10.0k    183ms    0.1%  18.3ΞΌs     0.00B    0.0%    0.00B
 2| Acceleration2                                     10.0k    168ms    0.1%  16.8ΞΌs     0.00B    0.0%    0.00B
 2| Gravity                                           10.0k    165ms    0.1%  16.5ΞΌs     0.00B    0.0%    0.00B
 2| DensityEpsi!                                      10.0k    108ms    0.1%  10.8ΞΌs     0.00B    0.0%    0.00B
 2| ρₙ⁺                                               10.0k   87.7ms    0.1%  8.77ΞΌs     0.00B    0.0%    0.00B
 2| LimitDensityAtBoundary!(ρₙ⁺)                      10.0k   46.4ms    0.0%  4.64ΞΌs     0.00B    0.0%    0.00B
 2| LimitDensityAtBoundary!(Density)                  10.0k   46.3ms    0.0%  4.63ΞΌs     0.00B    0.0%    0.00B
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────

@abraemer from the bottom of my heart, thank you so much for taking the time to look through the code. I have been chasing this GC and minor allocations previously for quite a few days now and such a simple change to the SimulationMetaData struct taught me so much and fixed the issue more or less completely.

Thanks a ton!

Kind regards

1 Like
10

So glad I could help you :smiley:

The final type instability caused by the type instability of the function LoadParticlesFromCSV. Because you load things from file the compiler cannot tell the types and so DF_FLUID.Rhop and DF_BOUND.Rhop are vectors of unknown type. You were saved from causing further type instability in the rest of the function because you copy these into an array of known type (Density).

To fix this, I would propose moving the loading of the files into the arguments of RunSimulation (and while we are doing this, we can also extract FloatType in a more idiomatic way). Here is a slightly modified start of RunSimulation with some comments:

function RunSimulation(;FluidCSV::String,
                        BoundCSV::String,
                        SimMetaData::SimulationMetaData{FloatType},
                        SimConstants::SimulationConstants,
                        (points,DF_FLUID,DF_BOUND)    = LoadParticlesFromCSV(FluidCSV,BoundCSV)
) where FloatType
    # FloatType - this  extracts the value inside of {} of the SimulationMetaData value
    # FloatType = typeof(SimMetaData).parameters[1]
    # we don't need this any more, if we extract the type directly from the args :)

    # Unpack the relevant simulation meta data
    @unpack HourGlass, SaveLocation, SimulationName, MaxIterations, OutputIteration, SilentOutput, ThreadsCPU = SimMetaData;

    # Unpack simulation constants
    @unpack ρ₀, dx, h, mβ‚€, Ξ±D, Ξ±, g, cβ‚€, Ξ³, dt, Ξ΄α΅©, CFL, Ξ·Β² = SimConstants

    # Load in the fluid and boundary particles. Return these points and both data frames
    # points,DF_FLUID,DF_BOUND    = LoadParticlesFromCSV(FluidCSV,BoundCSV)
    # this was moved to the args for type stability

    # Generate simulation data results array
    FinalResults = SimulationDataResults{3,FloatType}(NumberOfParticles = length(points))
    @unpack Kernel, KernelGradient, Density, Position, Acceleration, Velocity = FinalResults
    # Initialize Arrays
    Position .= deepcopy(points)
    # note that this allocates 2 tempory arrays (i think)
    Density  .= Array([DF_FLUID.Rhop;DF_BOUND.Rhop])
    # does this work as well?
    # Density  .= [DF_FLUID.Rhop;DF_BOUND.Rhop]
    # or maybe like this (no temp allocation but this doesn't matter here)
    # Density[1:length(DF_FLUID.Rhop)] .= DF_FLUID.Rhop
    # Density[length(DF_FLUID.Rhop)+1:end] .= DF_BOUND.Rhop

    # rest of the function
11

Hi again!

Thanks a ton, didn’t even know about FloatType inside of function signature with where clause, would work!

Got it, I ended up fixing density, by modifying the way I load particles in and preallocating results into a vector of type FloatType.

I was really confused though why something like:

Density .= FloatType.([Arr1;Arr2])

Would still give a type error? Is this due to it not knowing the types of Arr1 and Arr2 at compile time when it is from loading a CSV, so even forcefully converting to FloatType it cannot figure it out at compile time?

Kind regards

12

The type instability is coming from [Arr1; Arr2]. This expression is lowered to vcat(Arr1, Arr2) and since the element types of Arr1 and Arr2 are unknown this causes a dynamic dispatch and the return type is unknown. After that you convert it forcefully to FloatType by writing the values to Density which has a known type. (I am actually not sure if FloatType.(...) is strong enough to guarantee the type, it is anyways unecessary as the assignment attempts a conversion anyways).

So the type instability is not contagious in this case. It’s only during the setup that there are some instabilities but the rest of the code is perfectly type stable. So I don’t think this instability must be fixed necessarily :slight_smile:

1 Like