# How to stream a large random matrix to a file?

**URL:** <https://discourse.julialang.org/t/how-to-stream-a-large-random-matrix-to-a-file/52673>\
**Category:** General Usage\
**Tags:** question, io\
**Created:** [December 31, 2020, 10:27pm UTC](https://discourse.julialang.org/t/how-to-stream-a-large-random-matrix-to-a-file/52673 "2020-12-31T22:27:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [December 31, 2020, 10:27pm UTC](https://discourse.julialang.org/t/how-to-stream-a-large-random-matrix-to-a-file/52673/1 "2020-12-31T22:27:58Z")

</div>

The following attempts to stream a large random array to a file and read it with Mmap, but causes errors:

```julia
@testset "stream an m x n matrix to a file and read via Mmap" begin
    m=10
    n=10^5
    fn="/tmp/mmap.bin"
    s = open(fn, "w+")
    a1 = rand(m)
    for i in 1:n
        i == 1 ? write(s,a1) : write(s, rand(m))
    end
    close(s)
    @test isfile(fn)

    ss = open("/tmp/mmap.bin")
    A = Mmap.mmap(s, Matrix{Int}, (m,n))
    @test A[:,1] == a1
end

```

> **output with errors**
>
> ```julia
> stream an m x n matrix to a file and read via Mmap: Error During Test at REPL[286]:1
> Got exception outside of a @test
> SystemError: position: Bad file descriptor
> Stacktrace:
> [1] systemerror(::String, ::Int32; extrainfo::Nothing) at ./error.jl:168
> [2] #systemerror#48 at ./error.jl:167 [inlined]
> [3] systemerror at ./error.jl:167 [inlined]
> [4] position at ./iostream.jl:218 [inlined]
> [5] mmap(::IOStream, ::Type{Array{Int64,2}}, ::Tuple{Int64,Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Mmap/src/Mmap.jl:189
> [6] macro expansion at ./REPL[286]:13 [inlined]
> [7] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
> [8] top-level scope at ./REPL[286]:2
> [9] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1091
> [10] repleval(::Module, ::String, ::String) at /home/me/.vscode/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/repl.jl:105
> [11] (::VSCodeServer.var"#43#45"{Module,String,REPL.LineEditREPL,REPL.LineEdit.Prompt})() at /home/me/.vscode/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/repl.jl:84
> [12] with_logstate(::Function, ::Any) at ./logging.jl:408
> [13] with_logger at ./logging.jl:514 [inlined]
> [14] (::VSCodeServer.var"#42#44"{Module,String,REPL.LineEditREPL,REPL.LineEdit.Prompt})() at /home/me/.vscode/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/repl.jl:83
> [15] #invokelatest#1 at ./essentials.jl:710 [inlined]
> [16] invokelatest(::Any) at ./essentials.jl:709
> [17] macro expansion at /home/me/.vscode/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/eval.jl:27 [inlined]
> [18] (::VSCodeServer.var"#56#57")() at ./task.jl:356
> 
> ```

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [December 31, 2020, 10:52pm UTC](https://discourse.julialang.org/t/how-to-stream-a-large-random-matrix-to-a-file/52673/2 "2020-12-31T22:52:37Z")

</div>

> [@mkarikom](#):
>
> `A = Mmap.mmap(s, Matrix{Int}, (m,n))`

guess it should be `ss`, so:  
`A = Mmap.mmap(ss, Matrix{Int}, (m,n))`  
or  
`s = open("/tmp/mmap.bin")`  
not `ss`

Have a Happy New Year!!!  
Same to all of you Julia people !!!  
🍾 🍷 🍻 🥂

---

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [January 1, 2021, 7:43pm UTC](https://discourse.julialang.org/t/how-to-stream-a-large-random-matrix-to-a-file/52673/3 "2021-01-01T19:43:49Z")

</div>

Thanks for spotting that bug, @oheil. I think the issue was also that I had `open(... "w+")`. It finally worked when I switched to `open(... "w")`

---

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [January 1, 2021, 7:45pm UTC](https://discourse.julialang.org/t/how-to-stream-a-large-random-matrix-to-a-file/52673/4 "2021-01-01T19:45:58Z")

</div>

tldr: call open(… “w”) as in the [example provided by @stevengj here](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/7).

Thanks to @oheil for [spotting additional bugs](https://discourse.julialang.org/t/how-to-stream-a-large-random-to-a-file/52673/2).

The following works:

```julia
using Test,Mmap

@testset "stream an m x n matrix to a file and read via Mmap" begin
    m=10
    n=10^5
    fn="/tmp/mmap.bin"
    s = open(fn, "w")
    a1 = rand(m)
    for i in 1:n
        i == 1 ? write(s,a1) : write(s, rand(m))
    end
    close(s)
    @test isfile(fn)

    s = open("/tmp/mmap.bin")
    A = Mmap.mmap(s, Matrix{Float64}, (m,n))
    @test A[:,1] == a1
end

Test Summary: | Pass Total
stream an m x n matrix to a file and read via Mmap | 2 2
Test.DefaultTestSet("stream an m x n matrix to a file and read via Mmap", Any[], 2, false)

```