# Write/read multiple arrays of different types 300+ Million elements each. Brute force .jld seems stupid.

**URL:** <https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229>\
**Category:** Performance\
**Created:** [September 20, 2018, 12:25pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229 "2018-09-20T12:25:35Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mikkel-Holm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikkel-holm/32/5502_2.png) [@Mikkel-Holm](https://discourse.julialang.org/u/Mikkel-Holm)\
**Post date:** [September 20, 2018, 12:25pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/1 "2018-09-20T12:25:35Z")

</div>

I am working with a semi-large dataset consisting of multiple arrays of different types each of ~300 Mil indexes.  
Using the JLD package to load the data as .jld files takes around 60 minutes, which is simply too slow.

Writing/reading binary is takes only ~60 seconds, but then I have trouble working with strings.

As an example, how does one convert a large Array{UInt8,1} back to Array{String,1}, as illustrated below.  
**Write to binary**  
a = fill(“hello world”,300\*10^6); # a = fill(“hello world”, 10^5) if ram is an issue  
out = open(“test.bin”,“w”)  
write(out,a)  
close(out)  
**Read from binary**  
out = open(“test.bin”,“r”)  
a = read(out);

Then I am stuck at Char.(a[1:11]). How does one get back to the original array of “hello world”?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [September 20, 2018, 1:41pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/2 "2018-09-20T13:41:54Z")

</div>

`join(Char.(a[1:11])` returns `“hello world”, and

```julia
[join(Char.(a[1+11*i:11+11*i])) for i ∈ 0:100000-1]

```

returns your original array (in the `10^5` case above, RAM is an issue on this machine!

But it appears to me this is dependent on fixed width?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 20, 2018, 1:59pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/3 "2018-09-20T13:59:06Z")

</div>

To write variable length strings in binary, you have to encode the length in some way. Eg put an `Int16` in front (if your strings are shorter than 2^{15} bytes), etc. Then you can use the `String(::Vector{UInt8})` constructor after you have read that many bytes.

There are alternative solutions, such as mmapping the files etc, but the above is quite fast in practice.

---

<div class="post-metadata">

**Author:** ![Mikkel-Holm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikkel-holm/32/5502_2.png) [@Mikkel-Holm](https://discourse.julialang.org/u/Mikkel-Holm)\
**Post date:** [September 20, 2018, 2:15pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/4 "2018-09-20T14:15:52Z")

</div>

This only works for “hello world strings”. But useful to see the application of join to concatenate the character arrays into single strings.

---

<div class="post-metadata">

**Author:** ![Mikkel-Holm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikkel-holm/32/5502_2.png) [@Mikkel-Holm](https://discourse.julialang.org/u/Mikkel-Holm)\
**Post date:** [September 20, 2018, 2:21pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/5 "2018-09-20T14:21:35Z")

</div>

That is a great answer, but how would one go about implementing the `Int16` in front of strings option?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 20, 2018, 2:22pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/6 "2018-09-20T14:22:27Z")

</div>

`read(io, Int16)` and `write(io, Int16(...))` would take care of this.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 20, 2018, 2:22pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/7 "2018-09-20T14:22:40Z")

</div>

JLD2 loads 10\*10^7 strings in 3 seconds so should be faster than JLD for your dataset.

---

<div class="post-metadata">

**Author:** ![Mikkel-Holm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikkel-holm/32/5502_2.png) [@Mikkel-Holm](https://discourse.julialang.org/u/Mikkel-Holm)\
**Post date:** [September 20, 2018, 2:37pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/8 "2018-09-20T14:37:51Z")

</div>

@Tamas_Papp could you show a functioning example of how one would go about writing fx. [“1st string”,“2nd string”,“3rd string”] to binary and back to Array{String,1}.

It seems there is limited information on writing strings to binary. A simple example that can scale to 100+ million element arrays would be very useful for the community.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 20, 2018, 2:59pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/9 "2018-09-20T14:59:11Z")

</div>

Something like

```julia
# T is the type we use for string length, it has to be consistent between read and write
function write_string(io::IO, T::Type{<:Signed}, str::AbstractString)
    raw = codeunits(str)
    write(io, T(length(raw)))
    write(io, raw)
end

function read_string(io::IO, T::Type{<:Signed})
    len = read(io, T)
    String(read(io, len))
end

tmp = tempname()
str = "Ἐγχειρίδιον Ἐπικτήτου"

open(tmp, "w") do io
    write_string(io, Int16, str)
end

open(tmp, "r") do io
    read_string(io, Int16)
end

```

but I am sure someone can optimize this.

---

<div class="post-metadata">

**Author:** ![Mikkel-Holm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikkel-holm/32/5502_2.png) [@Mikkel-Holm](https://discourse.julialang.org/u/Mikkel-Holm)\
**Post date:** [September 20, 2018, 3:03pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/10 "2018-09-20T15:03:38Z")

</div>

Well its a start @Tamas_Papp🙂  
I will test it out on a 500+ million element string array, and get back with benchmarks.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 20, 2018, 3:04pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/11 "2018-09-20T15:04:48Z")

</div>

Here is another attempt

```julia
function write_strs(f, strings)
    open(f, "w") do io
        for str in strings
            write(io, UInt16(sizeof(str)))
            write(io, str)
        end
    end
end

function read_strs(f)
    strings = String[]
    open(f, "r") do io
        while !eof(io)
            l = read(io, UInt16)
            s = Base.StringVector(l)
            for i in 1:l
                s[i] = read(io, UInt8)
            end
            push!(strings, String(s))
        end
    end
    return strings
end

strings = ["aba", "cbc", "αβ", "longerstring...."]
write_strs("foo", strings)
read_strs("foo")

a = fill("hello world", 10^7)
write_strs("big", a)
@time b = read_strs("big")
@assert b==a

```

---

<div class="post-metadata">

**Author:** ![Mikkel-Holm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikkel-holm/32/5502_2.png) [@Mikkel-Holm](https://discourse.julialang.org/u/Mikkel-Holm)\
**Post date:** [September 20, 2018, 3:10pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/12 "2018-09-20T15:10:23Z")

</div>

@kristoffer.carlsson I will benchmark your attempt against @Tamas_Papp on 500+ million elements Array{String,1} and get back with results.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 20, 2018, 4:57pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/13 "2018-09-20T16:57:39Z")

</div>

JLD2 loads it in almost the same time as these handwritten functions so might be better just going with that. The filesize is a bit bigger though.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 21, 2018, 5:11am UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/14 "2018-09-21T05:11:24Z")

</div>

The problem with basing my workflow on JLD2 is that if it is broken, I can’t just fix it easily. This was a problem in practice with the transition to 1.0 for me, and since then I have been wary of using it.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 21, 2018, 2:09pm UTC](https://discourse.julialang.org/t/write-read-multiple-arrays-of-different-types-300-million-elements-each-brute-force-jld-seems-stupid/15229/15 "2018-09-21T14:09:22Z")

</div>

The 1.0 release should hopefully at least reduce that worry a bit.