# I can't help but think debugging memory allocations shouldn't be this hard

**URL:** <https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042>\
**Category:** General Usage\
**Created:** [October 17, 2023, 2:05am UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042 "2023-10-17T02:05:14Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 2:05am UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/1 "2023-10-17T02:05:14Z")

</div>

I do `julia --track-allocation=user b.jl `  
The relevant report here is:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/4/84b4ef6897a3406a0a0bc50737a6f02d3562ac01.png)  
and  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/d/dd6699d711bace2e6b5f3b03f14586501fc3363b.png)  
How does this make sense?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 17, 2023, 2:17am UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/2 "2023-10-17T02:17:45Z")

</div>

I have proposed removing `--track-lallocation` since it is prone to false-positives.

> <https://github.com/JuliaLang/julia/pull/48070>
>
> Prior to Julia 1.9 \`--track-allocation\` was one of the only ways to track where …allocations where coming from in Julia.
> The way it functions is to change the emitted LLVM IR to insert code that tracks allocations occuring.
> This often leads to false-positives since these statements interfere with optimization (in particular escape analysis)
> and @simonbyrne and I once spend a particular fun couple of days hunting down ghosts.
> 
> Since it changes the emitted code it also does not particular play well with package images.
> 
> I believe there is no remaining use-case for \`--track-allocation\` that would not be better served by using the allocation profiler,
> and I am proposing to remove of the \`--track-allocation\` feature in 1.10. 
> 
> Are there users of \`--track-allocation\` that would object? Are there use-cases where you have fund it particular useful that are not covered by the allocation profiler?

I strongly recommend using the allocation profiler instead to gain insights.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 3:18am UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/3 "2023-10-17T03:18:18Z")

</div>

I did use Profile.Alloc as well. It reported allocation as well, same spot.  
Can’t make sense of it. Btw: `@time` reported allocations too. So I don’t think it can all be false positives.

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 17, 2023, 3:25am UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/4 "2023-10-17T03:25:35Z")

</div>

Can you share the proof file? Maybe through [https://pprof.me](https://pprof.me)? Looking at what types are allocated might help with your question.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 2:22pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/5 "2023-10-17T14:22:19Z")

</div>

Thank you for your generous offer. The profiling reported here is consistent with that of `--track-allocation`:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/5/e/5e150418ebde0eab64b6695a343f59df9b02901b.png)  
Unfortunately, I cannot upload the file because it is over the 2MB limit.  
I made the file available here: [Dropbox - alloc-profile.pb.gz - Simplify your life](https://www.dropbox.com/scl/fi/jlf47v6t2czjr6qznuo6e/alloc-profile.pb.gz?rlkey=r2p8btwvvifk9cv2h8v6f8viq&dl=0)

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 2:27pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/6 "2023-10-17T14:27:14Z")

</div>

Is there some scenario where memory would be allocated _before_ the function `updatecsmat!` gets called? Because pprof does not even mention `updatecsmat!` among places where allocation occurs…

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 17, 2023, 3:34pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/7 "2023-10-17T15:34:43Z")

</div>

> [@PetrKryslUCSD](#):
>
> Unfortunately, I cannot upload the file because it is over the 2MB limit.

Ah yes, the issue is that PProf.jl didn’t store compressed data for the longest time. Version 3 will fix that.

Anyway:

Clicking on “Sample” gives you two views “Allocs” and “Size” the former is the number of allocations, the latter how much memory was used.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/6/a/6acc5eaf6b41107b3c28b266f400f980b4ca9f02.png)

This is the “size” view. We allocate two buffers one of size 76.34MB and one of 7.64MB; then we are also boxing a bunch of integers.

You can walk the chain up and you see that the buffer allocations are coming from sparse operations in `makematrix`.

Flipping over to `Allocs` we see the number of allocations:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/b/d/bd8c0843f222a686b9065ff35ccf1948ba3cb24f.jpeg)

So why would Julia box an integer on a call-site?

`code_typed`/`Cthulhu` should tell you that, but my guess is that we either needed to box the return value or that one of the arguments to the function is despecialized and we pass it as a box.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 3:51pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/8 "2023-10-17T15:51:41Z")

</div>

I have a lot to learn in the pprof analysis interface! I did notice the boxing: hence my question about `updatecsmat!` at the call site! Thanks!

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 5:13pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/9 "2023-10-17T17:13:02Z")

</div>

Hmm, I have used ` @code_warntype`, with the following result:

```julia
MethodInstance for FinEtools.FEMMBaseModule._bilform_diffusion_general(::FinEtoolsHeatDiff.FEMMHeatDiffModule.FEMMHeatDiff{FinEtools.IntegDomainModule.IntegDomain{FinEtools.FESetModule.FESetT3{Int64}, typeof(FinEtools.IntegDomainModule.otherdimensionunity), FinEtools.IntegRuleModule.TriRule}, FinEtoolsHeatDiff.MatHeatDiffModule.MatHeatDiff{Float64, typeof(FinEtoolsHeatDiff.MatHeatDiffModule.tangentmoduli!), typeof(FinEtoolsHeatDiff.MatHeatDiffModule.update!)}}, ::FinEtools.AssemblyModule.SysmatAssemblerSparseSymm{Int64, Float64, Vector{Float64}, Vector{Int64}}, ::FinEtools.NodalFieldModule.NodalField{Float64, Int64}, ::FinEtools.NodalFieldModule.NodalField{Float64, Int64}, ::FinEtools.DataCacheModule.DataCache{Matrix{Float64}, FinEtools.DataCacheModule.var"#_fillcache_constant!#1"})                                                                                                       
  from _bilform_diffusion_general(self::FEMM, assembler::A, geom::FinEtools.NodalFieldModule.NodalField{FT}, u::FinEtools.NodalFieldModule.NodalField{T}, cf::DC) where {FEMM<:AbstractFEMM, A<:AbstractSysmatAssembler, FT, T, DC<:DataCache} @ FinEtools.FEMMBaseModule C:\Users\pkonl\Documents\00WIP\FinEtools.jl\src\FEMMBaseModule.jl:1486                                                
Static Parameters                                                                                                               
  FEMM = FinEtoolsHeatDiff.FEMMHeatDiffModule.FEMMHeatDiff{FinEtools.IntegDomainModule.IntegDomain{FinEtools.FESetModule.FESetT3{Int64}, typeof(FinEtools.IntegDomainModule.otherdimensionunity), FinEtools.IntegRuleModule.TriRule}, FinEtoolsHeatDiff.MatHeatDiffModule.MatHeatDiff{Float64, typeof(FinEtoolsHeatDiff.MatHeatDiffModule.tangentmoduli!), typeof(FinEtoolsHeatDiff.MatHeatDiffModule.update!)}}                                                                                                                
  A = FinEtools.AssemblyModule.SysmatAssemblerSparseSymm{Int64, Float64, Vector{Float64}, Vector{Int64}}                        
  FT = Float64                                                                                                                  
  T = Float64                                                                                                                   
  DC = FinEtools.DataCacheModule.DataCache{Matrix{Float64}, FinEtools.DataCacheModule.var"#_fillcache_constant!#1"}             
Arguments                                                                                                                       
  #self#::Core.Const(FinEtools.FEMMBaseModule._bilform_diffusion_general)                                                       
  self::FinEtoolsHeatDiff.FEMMHeatDiffModule.FEMMHeatDiff{FinEtools.IntegDomainModule.IntegDomain{FinEtools.FESetModule.FESetT3{Int64}, typeof(FinEtools.IntegDomainModule.otherdimensionunity), FinEtools.IntegRuleModule.TriRule}, FinEtoolsHeatDiff.MatHeatDiffModule.MatHeatDiff{Float64, typeof(FinEtoolsHeatDiff.MatHeatDiffModule.tangentmoduli!), typeof(FinEtoolsHeatDiff.MatHeatDiffModule.update!)}}                                                                                                                 
  assembler::FinEtools.AssemblyModule.SysmatAssemblerSparseSymm{Int64, Float64, Vector{Float64}, Vector{Int64}}                 
  geom::FinEtools.NodalFieldModule.NodalField{Float64, Int64}                                                                   
  u::FinEtools.NodalFieldModule.NodalField{Float64, Int64}                                                                      
  cf::FinEtools.DataCacheModule.DataCache{Matrix{Float64}, FinEtools.DataCacheModule.var"#_fillcache_constant!#1"}              
Locals                                                                                                                          
  @_7::Union{Nothing, Tuple{Int64, Int64}}                                                                                      
  @_8::Int64                                                                                                                    
  @_9::Int64                                                                                                                    
  @_10::Int64                                                                                                                   
  @_11::Int64                                                                                                                   
  pc::Matrix{Float64}                                                                                                           
  w::Matrix{Float64}                                                                                                            
  gradNparams::Matrix{Matrix{Float64}}                                                                                          
  Ns::Matrix{Matrix{Float64}}                                                                                                   
  npts::Int64                                                                                                                   
  c_gradNT::Matrix{Float64}                                                                                                     
  RmTJ::Matrix{Float64}                                                                                                         
  elvec::Vector{Float64}                                                                                                        
  elmat::Matrix{Float64}                                                                                                        
  elmdim::Int64                                                                                                                 
  gradN::Matrix{Float64}                                                                                                        
  J::Matrix{Float64}                                                                                                            
  loc::Matrix{Float64}                                                                                                          
  dofnums::Vector{Int64}                                                                                                        
  ecoords::Matrix{Float64}                                                                                                      
  ndn::Int64                                                                                                                    
  nne::Int64                                                                                                                    
  fes::FinEtools.FESetModule.FESetT3{Int64}                                                                                     
  @_30::Union{Nothing, Tuple{Int64, Int64}}                                                                                     
  i::Int64                                                                                                                      
  j::Int64                                                                                                                      
  c::Matrix{Float64}                                                                                                            
  Jac::Float64                                                                                                                  
Body::SparseArrays.SparseMatrixCSC{Float64, Int64}                                                                              
1 ─ (fes = FinEtools.FEMMBaseModule.finite_elements(self))                                                               
│ %2 = FinEtools.FEMMBaseModule._buff_b(self, geom, u)::Core.PartialStruct(Tuple{Int64, Int64, Matrix{Float64}, Vector{Int64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}}, Any[Core.Const(3), Int64, Matrix{Float64}, Vector{Int64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}])                                                                                            
│ %3 = Base.indexed_iterate(%2, 1)::Core.Const((3, 2))                                                                      
│ (nne = Core.getfield(%3, 1))                                                                                         
│ (@_11 = Core.getfield(%3, 2))                                                                                        
│ %6 = Base.indexed_iterate(%2, 2, @_11::Core.Const(2))::Core.PartialStruct(Tuple{Int64, Int64}, Any[Int64, Core.Const(3)]) 
│ (ndn = Core.getfield(%6, 1))                                                                                         
│ (@_11 = Core.getfield(%6, 2))                                                                                        
│ %9 = Base.indexed_iterate(%2, 3, @_11::Core.Const(3))::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(4)])                                                                                                             
│ (ecoords = Core.getfield(%9, 1))                                                                                     
│ (@_11 = Core.getfield(%9, 2))                                                                                        
│ %12 = Base.indexed_iterate(%2, 4, @_11::Core.Const(4))::Core.PartialStruct(Tuple{Vector{Int64}, Int64}, Any[Vector{Int64}, 
Core.Const(5)])                                                                                                                 
│ (dofnums = Core.getfield(%12, 1))                                                                                    
│ (@_11 = Core.getfield(%12, 2))                                                                                       
│ %15 = Base.indexed_iterate(%2, 5, @_11::Core.Const(5))::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(6)])                                                                                                             
│ (loc = Core.getfield(%15, 1))                                                                                        
│ (@_11 = Core.getfield(%15, 2))                                                                                       
│ %18 = Base.indexed_iterate(%2, 6, @_11::Core.Const(6))::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(7)])                                                                                                             
│ (J = Core.getfield(%18, 1))                                                                                          
│ (@_11 = Core.getfield(%18, 2))                                                                                       
│ %21 = Base.indexed_iterate(%2, 7, @_11::Core.Const(7))::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(8)])                                                                                                             
│ (gradN = Core.getfield(%21, 1))                                                                                      
│ %23 = FinEtools.FEMMBaseModule._buff_e(self, geom, u, assembler)::Tuple{Int64, Matrix{Float64}, Vector{Float64}}           
│ %24 = Base.indexed_iterate(%23, 1)::Core.PartialStruct(Tuple{Int64, Int64}, Any[Int64, Core.Const(2)])                     
│ (elmdim = Core.getfield(%24, 1))                                                                                     
│ (@_10 = Core.getfield(%24, 2))                                                                                       
│ %27 = Base.indexed_iterate(%23, 2, @_10::Core.Const(2))::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(3)])                                                                                                            
│ (elmat = Core.getfield(%27, 1))                                                                                      
│ (@_10 = Core.getfield(%27, 2))                                                                                       
│ %30 = Base.indexed_iterate(%23, 3, @_10::Core.Const(3))::Core.PartialStruct(Tuple{Vector{Float64}, Int64}, Any[Vector{Float64}, Core.Const(4)])                                                                                                            
│ (elvec = Core.getfield(%30, 1))                                                                                      
│ %32 = FinEtools.FEMMBaseModule._buff_d(self, geom, u)::Tuple{Matrix{Float64}, Matrix{Float64}}                             
│ %33 = Base.indexed_iterate(%32, 1)::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(2)]) 
│ (RmTJ = Core.getfield(%33, 1))                                                                                       
│ (@_9 = Core.getfield(%33, 2))                                                                                        
│ %36 = Base.indexed_iterate(%32, 2, @_9::Core.Const(2))::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(3)])                                                                                                             
│ (c_gradNT = Core.getfield(%36, 1))                                                                                   
│ %38 = Base.getproperty(self, :integdomain)::FinEtools.IntegDomainModule.IntegDomain{FinEtools.FESetModule.FESetT3{Int64}, typeof(FinEtools.IntegDomainModule.otherdimensionunity), FinEtools.IntegRuleModule.TriRule}                                      
│ %39 = FinEtools.FEMMBaseModule.integrationdata(%38)::Tuple{Int64, Matrix{Matrix{Float64}}, Matrix{Matrix{Float64}}, Matrix{Float64}, Matrix{Float64}}                                                                                                      
│ %40 = Base.indexed_iterate(%39, 1)::Core.PartialStruct(Tuple{Int64, Int64}, Any[Int64, Core.Const(2)])                     
│ (npts = Core.getfield(%40, 1))                                                                                       
│ (@_8 = Core.getfield(%40, 2))                                                                                        
│ %43 = Base.indexed_iterate(%39, 2, @_8::Core.Const(2))::Core.PartialStruct(Tuple{Matrix{Matrix{Float64}}, Int64}, Any[Matrix{Matrix{Float64}}, Core.Const(3)])                                                                                             
│ (Ns = Core.getfield(%43, 1))                                                                                         
│ (@_8 = Core.getfield(%43, 2))                                                                                        
│ %46 = Base.indexed_iterate(%39, 3, @_8::Core.Const(3))::Core.PartialStruct(Tuple{Matrix{Matrix{Float64}}, Int64}, Any[Matrix{Matrix{Float64}}, Core.Const(4)])                                                                                             
│ (gradNparams = Core.getfield(%46, 1))                                                                                
│ (@_8 = Core.getfield(%46, 2))                                                                                        
│ %49 = Base.indexed_iterate(%39, 4, @_8::Core.Const(4))::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(5)])                                                                                                             
│ (w = Core.getfield(%49, 1))                                                                                          
│ (@_8 = Core.getfield(%49, 2))                                                                                        
│ %52 = Base.indexed_iterate(%39, 5, @_8::Core.Const(5))::Core.PartialStruct(Tuple{Matrix{Float64}, Int64}, Any[Matrix{Float64}, Core.Const(6)])                                                                                                             
│ (pc = Core.getfield(%52, 1))                                                                                         
│ %54 = FinEtools.FEMMBaseModule.size(elmat)::Tuple{Int64, Int64}                                                            
│ %55 = FinEtools.FEMMBaseModule.prod(%54)::Int64                                                                            
│ %56 = FinEtools.FEMMBaseModule.count(fes)::Int64                                                                           
│ %57 = (%55 * %56)::Int64                                                                                                   
│ %58 = FinEtools.FEMMBaseModule.nalldofs(u)::Int64                                                                          
│ %59 = FinEtools.FEMMBaseModule.nalldofs(u)::Int64                                                                          
│ FinEtools.FEMMBaseModule.startassembly!(assembler, %57, %58, %59)                                                    
│ %61 = FinEtools.FEMMBaseModule.eachindex(fes)::Core.PartialStruct(UnitRange{Int64}, Any[Core.Const(1), Int64])             
│ (@_7 = Base.iterate(%61))                                                                                            
│ %63 = (@_7 === nothing)::Bool                                                                                              
│ %64 = Base.not_int(%63)::Bool                                                                                              
└── goto #7 if not %64                                                                                                   
2 ┄ %66 = @_7::Tuple{Int64, Int64}                                                                                             
│ (i = Core.getfield(%66, 1))                                                                                          
│ %68 = Core.getfield(%66, 2)::Int64                                                                                         
│ %69 = ecoords::Matrix{Float64}                                                                                             
│ %70 = Base.getproperty(fes, :conn)::Vector{Tuple{Int64, Int64, Int64}}                                                     
│ %71 = Base.getindex(%70, i)::Tuple{Int64, Int64, Int64}                                                                    
│ FinEtools.FEMMBaseModule.gathervalues_asmat!(geom, %69, %71)                                                         
│ %73 = elmat::Matrix{Float64}                                                                                               
│ %74 = FinEtools.FEMMBaseModule.zero($(Expr(:static_parameter, 4)))::Core.Const(0.00000e+00)                                
│ FinEtools.FEMMBaseModule.fill!(%73, %74)                                                                             
│ %76 = (1:npts)::Core.PartialStruct(UnitRange{Int64}, Any[Core.Const(1), Int64])                                            
│ (@_30 = Base.iterate(%76))                                                                                           
│ %78 = (@_30 === nothing)::Bool                                                                                             
│ %79 = Base.not_int(%78)::Bool                                                                                              
└── goto #5 if not %79                                                                                                   
3 ┄ %81 = @_30::Tuple{Int64, Int64}                                                                                            
│ (j = Core.getfield(%81, 1))                                                                                          
│ %83 = Core.getfield(%81, 2)::Int64                                                                                         
│ %84 = loc::Matrix{Float64}                                                                                                 
│ %85 = J::Matrix{Float64}                                                                                                   
│ %86 = ecoords::Matrix{Float64}                                                                                             
│ %87 = Base.getindex(Ns, j)::Matrix{Float64}                                                                                
│ %88 = Base.getindex(gradNparams, j)::Matrix{Float64}                                                                       
│ FinEtools.FEMMBaseModule.locjac!(%84, %85, %86, %87, %88)                                                            
│ %90 = Base.getproperty(self, :integdomain)::FinEtools.IntegDomainModule.IntegDomain{FinEtools.FESetModule.FESetT3{Int64}, typeof(FinEtools.IntegDomainModule.otherdimensionunity), FinEtools.IntegRuleModule.TriRule}                                      
│ %91 = J::Matrix{Float64}                                                                                                   
│ %92 = loc::Matrix{Float64}                                                                                                 
│ %93 = Base.getproperty(fes, :conn)::Vector{Tuple{Int64, Int64, Int64}}                                                     
│ %94 = Base.getindex(%93, i)::Tuple{Int64, Int64, Int64}                                                                    
│ %95 = Base.getindex(Ns, j)::Matrix{Float64}                                                                                
│ (Jac = FinEtools.FEMMBaseModule.Jacobianvolume(%90, %91, %92, %94, %95))                                             
│ %97 = Base.getproperty(self, :mcsys)::FinEtools.CSysModule.CSys                                                            
│ %98 = loc::Matrix{Float64}                                                                                                 
│ %99 = J::Matrix{Float64}                                                                                                   
│ %100 = i::Int64                                                                                                             
│ FinEtools.FEMMBaseModule.updatecsmat!(%97, %98, %99, %100, j)                                                        
│ %102 = RmTJ::Matrix{Float64}                                                                                                
│ %103 = Base.getproperty(self, :mcsys)::FinEtools.CSysModule.CSys                                                            
│ %104 = FinEtools.FEMMBaseModule.csmat(%103)::Matrix{T} where T<:Number                                                      
│ FinEtools.FEMMBaseModule.mulCAtB!(%102, %104, J)                                                                     
│ %106 = fes::FinEtools.FESetModule.FESetT3{Int64}                                                                            
│ %107 = gradN::Matrix{Float64}                                                                                               
│ %108 = Base.getindex(gradNparams, j)::Matrix{Float64}                                                                       
│ FinEtools.FEMMBaseModule.gradN!(%106, %107, %108, RmTJ)                                                              
│ (c = (cf)(loc, J, i, j))                                                                                             
│ %111 = elmat::Matrix{Float64}                                                                                               
│ %112 = gradN::Matrix{Float64}                                                                                               
│ %113 = Jac::Float64                                                                                                         
│ %114 = Base.getindex(w, j)::Float64                                                                                         
│ %115 = (%113 * %114)::Float64                                                                                               
│ %116 = c::Matrix{Float64}                                                                                                   
│ FinEtools.FEMMBaseModule.add_gkgt_ut_only!(%111, %112, %115, %116, c_gradNT)                                         
│ (@_30 = Base.iterate(%76, %83))                                                                                      
│ %119 = (@_30 === nothing)::Bool                                                                                             
│ %120 = Base.not_int(%119)::Bool                                                                                             
└── goto #5 if not %120                                                                                                  
4 ─ goto #3                                                                                                              
5 ┄ FinEtools.FEMMBaseModule.complete_lt!(elmat)                                                                         
│ %124 = dofnums::Vector{Int64}                                                                                               
│ %125 = Base.getproperty(fes, :conn)::Vector{Tuple{Int64, Int64, Int64}}                                                     
│ %126 = Base.getindex(%125, i)::Tuple{Int64, Int64, Int64}                                                                   
│ FinEtools.FEMMBaseModule.gatherdofnums!(u, %124, %126)                                                               
│ FinEtools.FEMMBaseModule.assemble!(assembler, elmat, dofnums, dofnums)                                               
│ (@_7 = Base.iterate(%61, %68))                                                                                       
│ %130 = (@_7 === nothing)::Bool                                                                                              
│ %131 = Base.not_int(%130)::Bool                                                                                             
└── goto #7 if not %131                                                                                                  
6 ─ goto #2                                                                                                              
7 ┄ %134 = FinEtools.FEMMBaseModule.makematrix!(assembler)::SparseArrays.SparseMatrixCSC{Float64, Int64}                        
└── return %134                                                                                                          

```

In particular, I see

```julia
%100 = i::Int64                                                                                                             
│ FinEtools.FEMMBaseModule.updatecsmat!(%97, %98, %99, %100, j)  

```

There is no boxing as far as I can tell?

The julia code is:

```julia
function _bilform_diffusion_general(
    self::FEMM,
    assembler::A,
    geom::NodalField{FT},
    u::NodalField{T},
    cf::DC,
) where {FEMM<:AbstractFEMM,A<:AbstractSysmatAssembler,FT,T,DC<:DataCache}
    fes = finite_elements(self)
    nne, ndn, ecoords, dofnums, loc, J, gradN = _buff_b(self, geom, u)
    elmdim, elmat, elvec = _buff_e(self, geom, u, assembler)
    RmTJ, c_gradNT = _buff_d(self, geom, u)
    npts, Ns, gradNparams, w, pc = integrationdata(self.integdomain)
    startassembly!(assembler, prod(size(elmat)) * count(fes), nalldofs(u), nalldofs(u))
    for i in eachindex(fes) # Loop over elements
        gathervalues_asmat!(geom, ecoords, fes.conn[i])
        fill!(elmat, zero(T)) # Initialize element matrix
        for j in 1:npts # Loop over quadrature points
            locjac!(loc, J, ecoords, Ns[j], gradNparams[j])
            Jac = Jacobianvolume(self.integdomain, J, loc, fes.conn[i], Ns[j])
            updatecsmat!(self.mcsys, loc, J, i, j)
            mulCAtB!(RmTJ, csmat(self.mcsys), J) # local Jacobian matrix
            gradN!(fes, gradN, gradNparams[j], RmTJ)
            c = cf(loc, J, i, j)
            add_gkgt_ut_only!(elmat, gradN, (Jac * w[j]), c, c_gradNT)
        end # Loop over quadrature points
        complete_lt!(elmat)
        gatherdofnums!(u, dofnums, fes.conn[i])# retrieve degrees of freedom
        assemble!(assembler, elmat, dofnums, dofnums)# assemble symmetric matrix
    end # Loop over elements
    return makematrix!(assembler)
end

```

I know that it is an imposition to ask for an opinion, but perhaps you can see without needing to dig in?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 17, 2023, 5:28pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/10 "2023-10-17T17:28:36Z")

</div>

```julia
│ %97 = Base.getproperty(self, :mcsys)::FinEtools.CSysModule.CSys                                                            

```

This doesn’t look to be concrete — CSys has two parameters, no?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 5:53pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/11 "2023-10-17T17:53:49Z")

</div>

It does. But this is the definition of the `self`:

```julia
mutable struct FEMMBase{ID<:IntegDomain, CS<:CSys} <: AbstractFEMM
    integdomain::ID # domain data
    mcsys::CS # updater of the material orientation matrix
end

```

So, wouldn’t the type be known?

Also, I seem to have trouble with many allocations of integers (boxing).

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 17, 2023, 5:59pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/12 "2023-10-17T17:59:17Z")

</div>

That’s not the type of self:

> [@PetrKryslUCSD](#):
>
> `self::FinEtoolsHeatDiff.FEMMHeatDiffModule.FEMMHeatDiff{...}`

There mcsys isn’t parameterized.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 6:00pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/13 "2023-10-17T18:00:40Z")

</div>

OMG. I forgot to update here. Thanks!

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 17, 2023, 6:02pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/14 "2023-10-17T18:02:52Z")

</div>

Also telling is:

```julia
%103 = Base.getproperty(self, :mcsys)::FinEtools.CSysModule.CSys %104 = FinEtools.FEMMBaseModule.csmat(%103)::Matrix{T} where T<:Number                                                      

```

Since the eltype of the Matrix is unknown.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 6:08pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/15 "2023-10-17T18:08:44Z")

</div>

But: why did the profiling report boxing of integers?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 17, 2023, 6:13pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/16 "2023-10-17T18:13:25Z")

</div>

Because due to the type of `%100` being abstract,

```julia
 %97 = Base.getproperty(self, :mcsys)::FinEtools.CSysModule.CSys                                                            
│ %98 = loc::Matrix{Float64}                                                                                                 
│ %99 = J::Matrix{Float64}                                                                                                   
│ %100 = i::Int64                                                                                                             
│ FinEtools.FEMMBaseModule.updatecsmat!(%97, %98, %99, %100, j)          

```

You likely ended up with a different calling-convention that requires all arguments to be boxed.

You can normally see that with `@code_llvm`.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 6:19pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/17 "2023-10-17T18:19:05Z")

</div>

I see. But there was only a mention of the integers, nothing about boxing matrices. I will check out what you suggested. Thanks.

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 17, 2023, 11:12pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/18 "2023-10-17T23:12:55Z")

</div>

> [@PetrKryslUCSD](#):
>
> . But there was only a mention of the integers, nothing about boxing matrices. I will check out what you suggested. Thanks.

The type information of CSys being unavailable would cause a slower way of calling `updatecsmat!` which requires all the integers to be boxed. The array is already a boxed value.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 17, 2023, 11:29pm UTC](https://discourse.julialang.org/t/i-cant-help-but-think-debugging-memory-allocations-shouldnt-be-this-hard/105042/19 "2023-10-17T23:29:07Z")

</div>

Ah, I forgot about that!