How to stream a large random matrix to a file?

General Usage
,
1

The following attempts to stream a large random array to a file and read it with Mmap, but causes errors:

@testset "stream an m x n matrix to a file and read via Mmap" begin
    m=10
    n=10^5
    fn="/tmp/mmap.bin"
    s = open(fn, "w+")
    a1 = rand(m)
    for i in 1:n
        i == 1 ? write(s,a1) : write(s, rand(m))
    end
    close(s)
    @test isfile(fn)

    ss = open("/tmp/mmap.bin")
    A = Mmap.mmap(s, Matrix{Int}, (m,n))
    @test A[:,1] == a1
end
output with errors 
stream an m x n matrix to a file and read via Mmap: Error During Test at REPL[286]:1
  Got exception outside of a @test
  SystemError: position: Bad file descriptor
  Stacktrace:
   [1] systemerror(::String, ::Int32; extrainfo::Nothing) at ./error.jl:168
   [2] #systemerror#48 at ./error.jl:167 [inlined]
   [3] systemerror at ./error.jl:167 [inlined]
   [4] position at ./iostream.jl:218 [inlined]
   [5] mmap(::IOStream, ::Type{Array{Int64,2}}, ::Tuple{Int64,Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Mmap/src/Mmap.jl:189
   [6] macro expansion at ./REPL[286]:13 [inlined]
   [7] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
   [8] top-level scope at ./REPL[286]:2
   [9] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1091
   [10] repleval(::Module, ::String, ::String) at /home/me/.vscode/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/repl.jl:105
   [11] (::VSCodeServer.var"#43#45"{Module,String,REPL.LineEditREPL,REPL.LineEdit.Prompt})() at /home/me/.vscode/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/repl.jl:84
   [12] with_logstate(::Function, ::Any) at ./logging.jl:408
   [13] with_logger at ./logging.jl:514 [inlined]
   [14] (::VSCodeServer.var"#42#44"{Module,String,REPL.LineEditREPL,REPL.LineEdit.Prompt})() at /home/me/.vscode/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/repl.jl:83
   [15] #invokelatest#1 at ./essentials.jl:710 [inlined]
   [16] invokelatest(::Any) at ./essentials.jl:709
   [17] macro expansion at /home/me/.vscode/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/eval.jl:27 [inlined]
   [18] (::VSCodeServer.var"#56#57")() at ./task.jl:356
2

guess it should be ss, so:
A = Mmap.mmap(ss, Matrix{Int}, (m,n))
or
s = open("/tmp/mmap.bin")
not ss

Have a Happy New Year!!!
Same to all of you Julia people !!!
:champagne: :wine_glass: :beers: :clinking_glasses:

5 Likes
3

Thanks for spotting that bug, @oheil. I think the issue was also that I had open(... "w+"). It finally worked when I switched to open(... "w")

4

tldr: call open(… “w”) as in the example provided by @stevengj here.

Thanks to @oheil for spotting additional bugs.

The following works:

using Test,Mmap

@testset "stream an m x n matrix to a file and read via Mmap" begin
    m=10
    n=10^5
    fn="/tmp/mmap.bin"
    s = open(fn, "w")
    a1 = rand(m)
    for i in 1:n
        i == 1 ? write(s,a1) : write(s, rand(m))
    end
    close(s)
    @test isfile(fn)

    s = open("/tmp/mmap.bin")
    A = Mmap.mmap(s, Matrix{Float64}, (m,n))
    @test A[:,1] == a1
end

Test Summary:                                      | Pass  Total
stream an m x n matrix to a file and read via Mmap |    2      2
Test.DefaultTestSet("stream an m x n matrix to a file and read via Mmap", Any[], 2, false)
1 Like