# Writing an array too large to store in memory

**URL:** <https://discourse.julialang.org/t/writing-an-array-too-large-to-store-in-memory/9752>\
**Category:** Performance\
**Tags:** question, jld\
**Created:** [March 16, 2018, 4:25am UTC](https://discourse.julialang.org/t/writing-an-array-too-large-to-store-in-memory/9752 "2018-03-16T04:25:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dsweber2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dsweber2/32/3606_2.png) [@dsweber2](https://discourse.julialang.org/u/dsweber2)\
**Post date:** [March 16, 2018, 4:25am UTC](https://discourse.julialang.org/t/writing-an-array-too-large-to-store-in-memory/9752/1 "2018-03-16T04:25:23Z")

</div>

I am transforming a dataset, and the process of transforming it both takes a long time and takes up a large amount of memory. I really only need a single row of the matrix in memory at any given time, so is there some efficient way I can write each row to file?

Ideally, I would like to be able to do so using @parallel.

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [March 16, 2018, 7:03am UTC](https://discourse.julialang.org/t/writing-an-array-too-large-to-store-in-memory/9752/2 "2018-03-16T07:03:23Z")

</div>

By “matrix”, do you mean an array of fixed-size elements? If so, then have a look at [mmap](https://docs.julialang.org/en/latest/stdlib/Mmap/).

For writing one row at a time, it’s best if the array is stored in row-major order. If you need it to be column-major, then it’s probably more efficient to first store, say, 1024 rows in memory and then write that entire block to disk at once. (Of course `mmap` might already take care of this.) Rows within a block could be computed in parallel.

---

<div class="post-metadata">

**Author:** ![dsweber2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dsweber2/32/3606_2.png) [@dsweber2](https://discourse.julialang.org/u/dsweber2)\
**Post date:** [March 16, 2018, 12:53pm UTC](https://discourse.julialang.org/t/writing-an-array-too-large-to-store-in-memory/9752/3 "2018-03-16T12:53:30Z")

</div>

Yes exactly! Row-vs-column major isn’t a big deal to me, thanks for pointing that out. So going off the documentation, a protype would be

> A = rand(100,100)  
> s = open(“tmp.bin”, “w+”)
> 
> # We’ll write the dimensions of the array as the first two Ints in the file
> 
> write(s, size(A,1))  
> write(s, size(A,2))
> 
> # Now write the data
> 
> for i=1:100  
> write(s, A[:,i])  
> end  
> close(s)
> 
> s = open(“tmp.bin”) # default is read-only  
> m = read(s, Int)  
> n = read(s, Int)  
> A2 = Mmap.mmap(s, Matrix{Int}, (m,n))

I can’t seem to find a way to get JLD to write this way, even though it seems to have a way of using mmap.