# OutOfMemoryError()\_solved by sparse matrix
**URL:** https://discourse.julialang.org/t/outofmemoryerror-solved-by-sparse-matrix/35955
**Category:** General Usage
**Tags:** question
**Created:** [March 14, 2020, 3:18am UTC](https://discourse.julialang.org/t/outofmemoryerror-solved-by-sparse-matrix/35955 "2020-03-14T03:18:43Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@1634](https://discourse.julialang.org/u/1634)
#### Post date: [March 14, 2020, 3:18am UTC](https://discourse.julialang.org/t/outofmemoryerror-solved-by-sparse-matrix/35955/1 "2020-03-14T03:18:43Z")
Hi community,
```julia
julia> @time legend, t_avn_mtx, t_se_mtx, eta_avn_mtx, eta_se_mtx, fix_M_mtx =
onedim_xdispersal_yfix_nDemesseries(N,nDemes_vec,twoNmu,s0,twoNDememig_vec,M,T,Gauss, nu)
t_fix_vecArray{Any,1}(10,)
t_fix_vecArray{Any,1}(10,)
t_fix_vecArray{Any,1}(10,)
t_fix_vecArray{Any,1}(10,)
t_fix_vecArray{Any,1}(10,)
ERROR: OutOfMemoryError()
Stacktrace:
[1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./task.jl:517
[2] wait() at ./task.jl:592
[3] wait(::Base.GenericCondition{SpinLock}) at ./condition.jl:104
[4] wait(::Task) at ./task.jl:191
[5] macro expansion at ./threadingconstructs.jl:75 [inlined]
[6] onedim_xdispersal_yfix_nDemesseries(::Float64, ::Array{Int64,1}, ::Int64, ::Float64, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64, ::Int64) at /home/joejyn/Documents/Julia/twoNDememig/4_4_figjoejyn_series_threads.jl:101
[7] top-level scope at util.jl:156
```
And the 156line traced back to the first line as below, while the function inside the block is the main computation part:
```julia
@threads for i in 1:len_nDemes # line156
for j in 1:len_mig
t_avn, t_se, eta_avn, eta_se, fix_M=
num_origin_1d_fix(N,nDemes_vec[i],twoNmu,s0,twoNDememig_vec[j],M,T,Gauss, nu) t_avn_mtx[j,i]=t_avn
t_se_mtx[j,i]=t_se
eta_avn_mtx[j,i]=eta_avn
eta_se_mtx[j,i]=eta_se
fix_M_mtx[j,i]=fix_M
end
end
```
The devince information is
```julia
Julia Version 1.2.0
Commit c6da87ff4b (2019-08-20 00:03 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, broadwell)
Environment:
JULIA_PKG_SERVER = https://pkg.juliacomputing.com/
JULIA_PKG_TOKEN_PATH = /home/joejyn/.juliapro/token.toml
JULIA_DEPOT_PATH = /home/joejyn/.juliapro/JuliaPro_v1.2.0-2:/usr/local/julia/JuliaPro-1.2.0-2/Julia/local/share/julia:/usr/local/julia/JuliaPro-1.2.0-2/Julia/share/julia
```
What could the cause be?
Just in case it’s due to the array type:
[https://stackoverflow.com/questions/49868567/outofmemoryerror-in-](https://stackoverflow.com/questions/49868567/outofmemoryerror-in-)
JuliaDB as suggested solution in this post seems to serve in data management instead of an array that I want (the name of each column not specified).
On another post, the instruction of the memory-mapping is suggested, From [Memory-mapped I/O · The Julia Language](https://docs.julialang.org/en/v1/stdlib/Mmap/), the instruction is:
```julia
julia> B = Mmap.mmap(io, BitArray, (25,30000));
julia> B[3, 4000] = true;
```
I may not get the array to assign to B due to the outofmemory problem?
Could anyone point a way out?
---
### Author:  [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)
#### Post date: [March 14, 2020, 6:40am UTC](https://discourse.julialang.org/t/outofmemoryerror-solved-by-sparse-matrix/35955/2 "2020-03-14T06:40:34Z")
There is not enough given for me to recreate the problem.
For instance, what happens in `num_origin_1d_fix` is not known. It is always easiest to track down a problem when the context (lines of code) has been stripped down as much as possible, while still triggering the error.
Just guessing, your out of memory error may happen because there is something that could be better structured internally – leaking a great deal of memory inadvertantly is not usually a problem with Julia code; however you may be generating many copies of arrays for which only one or a few are utilized. And it is certainly possible that you have created dependencies among some chain[s] of them that is preventing the Garbage Collector from culling. Alternatively, you may be spawning many more threads than intended or requiring each to have very large local memory. This is all speculation, though.
> [@Please read: make it easier to help you](https://discourse.julialang.org/t/psa-make-it-easier-to-help-you/14757):
>
> Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced. This public service announcement (PSA) outlines best practices when asking for help. Following these points makes it easier for us to help you and more likely you’ll get a prompt, useful answer. Keywords are highlighted to make it easier to refer to specific points. Choose a descriptive title that captures the key part of your question, eg “plots with multiple axes” instead of …
---
### Author:  [@1634](https://discourse.julialang.org/u/1634)
#### Post date: [March 14, 2020, 10:12am UTC](https://discourse.julialang.org/t/outofmemoryerror-solved-by-sparse-matrix/35955/3 "2020-03-14T10:12:51Z")
> [@JeffreySarnoff](#):
>
> however you may be generating many copies of arrays for which only one or a few are utilized.
This is exactly right as I kept adding rows with few columns of then containing data.
Though I don’t comprehend what’s recommended in this:
> [@JeffreySarnoff](#):
>
> have created dependencies among some chain[s] of them that is preventing the Garbage Collector from culling
Based on your suggestion, I would try:
1.generating data and plot using Sparse matrix
2.delete the @threads call
3.profiling
Thank you~
---
### Author:  [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)
#### Post date: [March 14, 2020, 10:39am UTC](https://discourse.julialang.org/t/outofmemoryerror-solved-by-sparse-matrix/35955/4 "2020-03-14T10:39:41Z")
“I kept adding rows where few of the columns contained data”
[Sparse matricies (click for docs)](https://docs.julialang.org/en/latest/stdlib/SparseArrays/#Sparse-Vector-and-Matrix-Constructors-1) sound appropriate.
Another [good practice](https://docs.julialang.org/en/v1/manual/performance-tips/) is to preallocate (and zero) the size needed then put values into it, rather than continually adding onto it.
“what’s recommended in this” (about Garbage collection)
Try running it with smaller data matricies first; when it works run it with the larger data matricies.
I agree about the threads; there is little reason to introduce threading before all the other aspects of the project have been worked out.
_happy to be of help_