# File for mmap with an unknown number of structs

**URL:** <https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380>\
**Category:** General Usage\
**Tags:** question\
**Created:** [October 10, 2017, 4:18pm UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380 "2017-10-10T16:18:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 10, 2017, 4:18pm UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380/1 "2017-10-10T16:18:51Z")

</div>

Consider a struct (for MWE)

```julia
struct Foo{T1, T2}
    id::T1
    typ::T2
    start::Date
    stop::Date
end

```

My algorithm works in two stages:

1. I am generating a large number of these with fixed `T1` and `T2`, want to write that out into a file (this pass is write-only, sequential, no need to look up previous values), but _the dimension is unknown ex ante_,
2. I want to `Mmap.mmap` this file to a `Vector{Foo{T1,T2}}` and work on it; this pass is read-only, random access.

I have read the docstring for `mmap`, and considered using `write`, but apparently `write` does not support custom `struct`s. Should I just write a method for `Foo` above, `write`ing out the fields sequentially, eg

```julia
function write(io::IO, x::Foo)
    write(io, x.id, x.typ, x.start, x.stop)
end

```

or is there anything that requires special care for reading later with `mmap`? I am thinking about memory alignment especially, eg if `T1==Int` and `T2=Int8`.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 11, 2017, 8:36am UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380/2 "2017-10-11T08:36:00Z")

</div>

Update: of course `write(io, ...)` does not work in general, because of padding. I made it work with `reinterpret` and also `unsafe_write`, but advice on whether I am really doing the right thing would be appreciated. Self-contained MWE below.

```julia
struct Record{T1, T2}
    id::T1
    ix::T2
    date::Date
end

random_Date() = Date(rand(1900:2100), rand(1:12), rand(1:28))
random_record(T1, T2) = Record(rand(T1), rand(T2), random_Date())

T1 = Int16
T2 = Int8
path = "/tmp/test"

## random array
x = [random_record(T1, T2) for _ in 1:200]

############################################################
# write elementwise -- this DOES NOT WORK, because of padding
############################################################

@generated function write_struct(io, x)
    @assert isbits(x)
    if Base.isstructtype(x)
        Expr(:block, [:(write_struct(io, x.$fn)) for fn in fieldnames(x)]...)
    else
        :(write(io, x))
    end
end

io = open(path, "w")
for elt in x
    write_struct(io, elt)
end
close(io)

## read
io = open(path, "r")
y = Mmap.mmap(io, Vector{Record{T1, T2}}, (length(x),))

x == y # false, because of:
sizeof(x[1]) # 11
sizeof(y[1]) # 16

############################################################
# write with reinterpret -- this does work
############################################################

function write_reinterpret(io, x)
    @assert isbits(typeof(x))
    write(io, reinterpret(UInt8, [x]))
end

io = open(path, "w")
for elt in x
    write_reinterpret(io, elt)
end
close(io)

## read
io = open(path, "r")
y = Mmap.mmap(io, Vector{Record{T1, T2}}, (length(x),))
x == y # true

############################################################
# write with reinterpret -- this DOES WORK
############################################################

function write_struct_unsafe(io, x::T) where {T}
    @assert isbits(T)
    unsafe_write(io, Ref(x), sizeof(x)) # am I using this the right way?
end

io = open(path, "w")
for elt in x
    write_struct_unsafe(io, elt)
end
close(io)

## read
io = open(path, "r")
y = Mmap.mmap(io, Vector{Record{T1, T2}}, (length(x),))
x == y # true

```

Related discussion:  
[https://github.com/JuliaLang/julia/issues/10140](https://github.com/JuliaLang/julia/issues/10140)  
[https://github.com/JuliaLang/julia/issues/10354](https://github.com/JuliaLang/julia/issues/10354)

---

<div class="post-metadata">

**Author:** ![ihnorton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihnorton/32/26_2.png) [@ihnorton](https://discourse.julialang.org/u/ihnorton)\
**Post date:** [October 11, 2017, 12:01pm UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380/3 "2017-10-11T12:01:51Z")

</div>

Have you seen [https://github.com/Keno/StructIO.jl](https://github.com/Keno/StructIO.jl)? It handles packing and alignment calculations, and is used for e.g. inspection of big complex structs defined by various [object file formats](https://github.com/staticfloat/ObjectFile.jl).

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 11, 2017, 12:31pm UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380/4 "2017-10-11T12:31:03Z")

</div>

I may misunderstand, but my impression is that `StructIO.jl` solves a different problem: packing to a well-defined, system-independent format. Whereas I need a format that I can `mmap` to directly.

My current working hypothesis is that for `isbits` types, Julia [follows](https://docs.julialang.org/en/stable/manual/calling-c-and-fortran-code/#Struct-Type-correspondences-1) the [padding conventions](http://www.catb.org/esr/structure-packing/) of `C`, and `unsafe_write` works with that, so I should be OK, especially if I use `sizeof` since that takes padding into account.

Nevertheless, if would be very helpful is someone experienced with `mmap` could look at my code and tell me if I am doing something bogus. I have no prior experience with `mmap`.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 11, 2017, 3:12pm UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380/5 "2017-10-11T15:12:00Z")

</div>

> [@Tamas\_Papp](#):
>
> unsafe\_write(io, Ref(x), sizeof(x)) # am I using this the right way?

Yes.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 11, 2017, 3:13pm UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380/6 "2017-10-11T15:13:01Z")

</div>

> [@Tamas\_Papp](#):
>
> Julia follows the padding conventions of C

Yes. Julia structs are ABI compatible with C, except when there’s bug (currently the only ones I’m aware of are (SIMD) vectors and Int128 on x86)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 11, 2017, 3:22pm UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380/7 "2017-10-11T15:22:48Z")

</div>

Thanks! Related: how to use `Mmap.Anonymous`? This gives an error:

```julia
julia> Mmap.mmap(Mmap.Anonymous(), Vector{UInt8}, (10,))
ERROR: MethodError: no method matching position(::Base.Mmap.Anonymous)
Closest candidates are:
  position(::Base.Filesystem.File) at filesystem.jl:203
  position(::Base.Libc.FILE) at libc.jl:91
  position(::IOStream) at iostream.jl:69
  ...
Stacktrace:
 [1] mmap(::Base.Mmap.Anonymous, ::Type{Array{UInt8,1}}, ::Tuple{Int64}) at ./mmap.jl:102

```

Is this a bug? I tried looking for examples in the tests, but could not find one.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 11, 2017, 3:45pm UTC](https://discourse.julialang.org/t/file-for-mmap-with-an-unknown-number-of-structs/6380/8 "2017-10-11T15:45:11Z")

</div>

`Mmap.mmap(Mmap.Anonymous(), Vector{UInt8}, (10,), 0)` works. The default value for this argument (offset) seems to be problematic and it does seem like a bug.