If you can manage to produce smaller, self-contained examples, the discussion could be more focused. As it is right now, there is just a large piece of code that seems to have type instabilities, and allocate a lot of arrays. Then yeah, there will be a lot of memory allocations.
1 Like
Agree @kristoffer.carlsson, my previous comment has a self-contained example, the smallest possible in this framework.
The good news is that at least I got pmap to work now, and these slow down is spread across processes (in a cluster). 
I think you misunderstand “self contained”. With that, I mean that that the example has everything that is not essential for the problem removed. Distill it until the very core, so to say. Then we can have a discussion about the Julia principles to follow to reduce allocations and where the misunderstanding is, without worrying about the implementation.
You can of course also ask the question “Could someone find the type instabilities and unnecessary allocations in my package”. At least to me, that sounds less like Julia help and more like contributing to the package.
The reason why I say this is that I believe you will in general get much more help if you manage to ask pure Julia questions instead of “improve my package” -questions.
5 Likes
I have to agree with Kistoffer, this is a very large codebase to investigate here. These function calls are all type-unstable: (the last three are good points of focus)
1 trace like GeoStatsBase.valid(3×4 GeoDataFrame (x and y), :precipitation) => false
3 traces like GeoStats.solve_single(2D SimulationProblem (conditional), :precipitation, SeqGaussSim solver, GeoStats.SimpleMapper{GeoStats.RegularGrid{Float64,2}}(Dict{Symbol,Dict{Int64,#s27} where #s27}(Pair{Symbol,Dict{Int64,#s27} where #s27}(:precipitation, Dict{Int64,Float64}())))) => false
3 traces like GeoStatsBase.coordinates(3×4 GeoDataFrame (x and y)) => false
3 traces like GeoStats.mapping(GeoStats.SimpleMapper{GeoStats.RegularGrid{Float64,2}}(Dict{Symbol,Dict{Int64,#s27} where #s27}(Pair{Symbol,Dict{Int64,#s27} where #s27}(:precipitation, Dict{Int64,Float64}()))), :precipitation) => false
297 traces like GeoStats.fit!(<OrdinaryKriging object>, [9.0; 1.0], [0.158927]) => false
297 traces like GeoStats.estimate(<OrdinaryKriging object>, [3.0, 3.0]) => false
297 traces like GeoStats.combine(<GeoStats.OrdinaryKrigingWeights object>) => false
If you just want to reduce allocation, focus on type-stability inside your inner loop. Your solve_single function is very long, and if you introduce a function barrier before the loop, then it might also improve performance/allocation significantly.
2 Likes
Thank you @kristoffer.carlsson and @cstjean, I was wrapping up yesterday with a few other items, but I will try to write simple examples today highlighting the issues I am having. I will create a separate thread for each of them to make it more searchable and visible to others.
1 Like