OutOfMemoryError()_solved by sparse matrix

General Usage
1

Hi community,

julia> @time legend, t_avn_mtx, t_se_mtx, eta_avn_mtx, eta_se_mtx, fix_M_mtx =
         onedim_xdispersal_yfix_nDemesseries(N,nDemes_vec,twoNmu,s0,twoNDememig_vec,M,T,Gauss, nu)
t_fix_vecArray{Any,1}(10,)
t_fix_vecArray{Any,1}(10,)
t_fix_vecArray{Any,1}(10,)
t_fix_vecArray{Any,1}(10,)
t_fix_vecArray{Any,1}(10,)
ERROR: OutOfMemoryError()
Stacktrace:
 [1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./task.jl:517
 [2] wait() at ./task.jl:592
 [3] wait(::Base.GenericCondition{SpinLock}) at ./condition.jl:104
 [4] wait(::Task) at ./task.jl:191
 [5] macro expansion at ./threadingconstructs.jl:75 [inlined]
 [6] onedim_xdispersal_yfix_nDemesseries(::Float64, ::Array{Int64,1}, ::Int64, ::Float64, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64, ::Int64) at /home/joejyn/Documents/Julia/twoNDememig/4_4_figjoejyn_series_threads.jl:101
 [7] top-level scope at util.jl:156

And the 156line traced back to the first line as below, while the function inside the block is the main computation part:

 @threads for i in 1:len_nDemes # line156
          for j in 1:len_mig
            t_avn, t_se, eta_avn, eta_se, fix_M=
            num_origin_1d_fix(N,nDemes_vec[i],twoNmu,s0,twoNDememig_vec[j],M,T,Gauss, nu)            t_avn_mtx[j,i]=t_avn
            t_se_mtx[j,i]=t_se
            eta_avn_mtx[j,i]=eta_avn
            eta_se_mtx[j,i]=eta_se
            fix_M_mtx[j,i]=fix_M
         end
    end

The devince information is

Julia Version 1.2.0
Commit c6da87ff4b (2019-08-20 00:03 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_PKG_SERVER = https://pkg.juliacomputing.com/
  JULIA_PKG_TOKEN_PATH = /home/joejyn/.juliapro/token.toml
  JULIA_DEPOT_PATH = /home/joejyn/.juliapro/JuliaPro_v1.2.0-2:/usr/local/julia/JuliaPro-1.2.0-2/Julia/local/share/julia:/usr/local/julia/JuliaPro-1.2.0-2/Julia/share/julia

What could the cause be?

Just in case it’s due to the array type:
https://stackoverflow.com/questions/49868567/outofmemoryerror-in-
JuliaDB as suggested solution in this post seems to serve in data management instead of an array that I want (the name of each column not specified).

On another post, the instruction of the memory-mapping is suggested, From Memory-mapped I/O · The Julia Language, the instruction is:

julia> B = Mmap.mmap(io, BitArray, (25,30000));
julia> B[3, 4000] = true;

I may not get the array to assign to B due to the outofmemory problem?
Could anyone point a way out?

2

There is not enough given for me to recreate the problem.
For instance, what happens in num_origin_1d_fix is not known. It is always easiest to track down a problem when the context (lines of code) has been stripped down as much as possible, while still triggering the error.

Just guessing, your out of memory error may happen because there is something that could be better structured internally – leaking a great deal of memory inadvertantly is not usually a problem with Julia code; however you may be generating many copies of arrays for which only one or a few are utilized. And it is certainly possible that you have created dependencies among some chain[s] of them that is preventing the Garbage Collector from culling. Alternatively, you may be spawning many more threads than intended or requiring each to have very large local memory. This is all speculation, though.

1 Like
3

This is exactly right as I kept adding rows with few columns of then containing data.
Though I don’t comprehend what’s recommended in this:

Based on your suggestion, I would try:
1.generating data and plot using Sparse matrix
2.delete the @threads call
3.profiling
Thank you~

4

“I kept adding rows where few of the columns contained data”
Sparse matricies (click for docs) sound appropriate.

Another good practice is to preallocate (and zero) the size needed then put values into it, rather than continually adding onto it.

“what’s recommended in this” (about Garbage collection)
Try running it with smaller data matricies first; when it works run it with the larger data matricies.

I agree about the threads; there is little reason to introduce threading before all the other aspects of the project have been worked out.

happy to be of help