# Mmap and memory alignment

**URL:** <https://discourse.julialang.org/t/mmap-and-memory-alignment/20665>\
**Category:** General Usage\
**Created:** [February 11, 2019, 2:57pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665 "2019-02-11T14:57:32Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [February 11, 2019, 2:57pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/1 "2019-02-11T14:57:33Z")

</div>

I have binary files ([IAEA phase space files](https://www-nds.iaea.org/phsp/phsp.htmlx)) which contain large contiguous lists of particles. And I want to mmap these files.

Here is a simplified variant of my issue: A particle consists of 5 bytes, where the first byte encodes the type `UInt8` and the other bytes encode the energy `Float32`. Now one way to mimic this in julia would be:

```julia
struct Particle
    type::UInt8
    energy::Float32
end

```

However there is a problem. `sizeof(Particle) = 8`, while the particles on disk are only 5 bytes. This means I cannot `mmap` them. Are there recommended solutions to this?  
My idea was to instead do:

```julia
struct Particle
    bytes::NTuple{5,UInt8}
end

```

and then `reinterpret` chunks of the bytes field. However one problem I have is that `reinterpret` is quite picky:

```julia
bytes = (0x0,0x0,0x0,0x0,0x0)
reinterpret(Float32, bytes[2:5])

```

```julia
bitcast: expected primitive type value for second argument

```

So is this generally a good approach? If so, how to properly extract the `energy` field in the above example?

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [February 11, 2019, 3:16pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/2 "2019-02-11T15:16:36Z")

</div>

I’m not sure what the best approach is in julia 1.0.

We were dealing with the same issue trying to mmap structs that didn’t align [in LasIO](https://github.com/visr/LasIO.jl/pull/10/files#diff-4011852eb38665daa0047e5f64e1be75R6). We ended up memory mapping a `Vector{UInt8}`, and putting that into our own `AbstractVector` which could be used to retrieve individual structs. This was mostly based on the approach in [GitHub - JuliaArrays/UnalignedVectors.jl: Create arrays from memory buffers that lack appropriate alignment](https://github.com/JuliaArrays/UnalignedVectors.jl).

Only UnalignedVectors.jl package dates from 0.6, and has not been updated to 1.0. Probably this has made it easier now:

> `reinterpret` now works on any AbstractArray using the new `ReinterpretArray` type.  
> This supersedes the old behavior of reinterpret on Arrays. As a result, reinterpreting  
> arrays with different alignment requirements (removed in 0.6) is once again allowed ([https://github.com/JuliaLang/julia/pull/23750](https://github.com/JuliaLang/julia/pull/23750)).

---

<div class="post-metadata">

**Author:** ![zgornel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zgornel/32/217487_2.png) [@zgornel](https://discourse.julialang.org/u/zgornel)\
**Post date:** [February 11, 2019, 4:29pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/4 "2019-02-11T16:29:06Z")

</div>

One solution is to create a primitive type `Iaea40` covering the 40 bits of information and manually implementing the interface needed to reinterpret it as a `Particle` (with which you do the processing). So, from memory-mapped `Matrix{Iaea40}` the getters and setters would do the conversion to and from `Particle` i.e. `reinterpret(Particle, Iaea40)`. I never tried such an approach so this is somewhat theoretical but it should work 🙂

---

<div class="post-metadata">

**Author:** ![halleysfifthinc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/halleysfifthinc/32/206280_2.png) [@halleysfifthinc](https://discourse.julialang.org/u/halleysfifthinc)\
**Post date:** [February 11, 2019, 6:07pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/5 "2019-02-11T18:07:02Z")

</div>

While

> [@jw3126](#):
>
> ```julia
> bytes = (0x0,0x0,0x0,0x0,0x0)
> reinterpret(Float32, bytes[2:5])
> 
> ```

doesn’t work, this does:

```julia
julia> bytes2 = [0x0, 0x1, 0x2, 0x3, 0x4]
5-element Array{UInt8,1}:
 0x00
 0x01
 0x02
 0x03
 0x04

julia> reinterpret(Float32, bytes2[2:5])
1-element reinterpret(Float32, ::Array{UInt8,1}):
 1.5399896e-36

```

and if you are looking to avoid allocations/want your `Particle` type to be a `bitstype`, you could use an `SVector` instead:

```julia
julia> using StaticArrays

julia> bytes3 = @SVector [0x0, 0x1, 0x2, 0x3, 0x4]
5-element SArray{Tuple{5},UInt8,1,5}:
 0x00
 0x01
 0x02
 0x03
 0x04

julia> reinterpret(Float32, bytes3[2:5])
1-element reinterpret(Float32, ::Array{UInt8,1}):
 1.5399896e-36

```

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [February 11, 2019, 9:47pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/6 "2019-02-11T21:47:30Z")

</div>

@zgornel This is a nice idea. Unfortunatelly in my case, the size of Particle may vary depending on some header file, so this is not really an option.

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [February 11, 2019, 9:53pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/7 "2019-02-11T21:53:59Z")

</div>

Unfortunately this has very bad performance:

```julia
using StaticArrays, BenchmarkTools

x = @SVector zeros(UInt8, 4)
@benchmark reinterpret($Float32, $x)

```

```julia
BenchmarkTools.Trial: 
  memory estimate: 32 bytes
  allocs estimate: 2
  --------------
  minimum time: 69.666 ns (0.00% GC)
  median time: 75.958 ns (0.00% GC)
  mean time: 85.552 ns (10.43% GC)
  maximum time: 59.599 μs (99.82% GC)
  --------------
  samples: 10000
  evals/sample: 974

```

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [February 11, 2019, 10:57pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/8 "2019-02-11T22:57:37Z")

</div>

> [@jw3126](#):
>
> So is this generally a good approach? If so, how to properly extract the `energy` field in the above example?

The easiest option is just define your own getters:

```julia
struct Particle
    bytes::NTuple{5,UInt8}
end

particletype(p::Particle) = p.bytes[1]
function energy(p::Particle)
    u = (UInt32(p.bytes[2]) << 24) | (UInt32(p.bytes[3]) << 16) | (UInt32(p.bytes[4]) << 8) | UInt32(p.bytes[5])
    reinterpret(Float32, u)
end

```

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [February 11, 2019, 11:01pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/9 "2019-02-11T23:01:06Z")

</div>

> [@jw3126](#):
>
> Unfortunatelly in my case, the size of Particle may vary depending on some header file

This shouldn’t be a problem as long as the particle size for one mmaped region stays constant, right?

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [February 11, 2019, 11:07pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/10 "2019-02-11T23:07:35Z")

</div>

@Sukera right, this was in response to

> [@Mmap and memory alignment](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/4):
>
> One solution is to create a primitive type Iaea40 covering the 40 bits of information and manually implementing the interface needed to reinterpret it as a Particle (with which you do the processing). So, from memory-mapped Matrix{Iaea40} the getters and setters would do the conversion to and from Particle i.e. reinterpret(Particle, Iaea40). I never tried such an approach so this is somewhat theoretical but it should work slight_smile

where the size is hard coded once and for all (I like the approach otherwise).

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [February 11, 2019, 11:17pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/11 "2019-02-11T23:17:46Z")

</div>

Nice! I was worried, that performance could be bad, but it is not:

```julia
using BenchmarkTools
struct Particle
    bytes::NTuple{5,UInt8}
end

particletype(p::Particle) = p.bytes[1]
function energy(p::Particle)
    u = (UInt32(p.bytes[2]) << 24) | (UInt32(p.bytes[3]) << 16) | (UInt32(p.bytes[4]) << 8) | UInt32(p.bytes[5])
    reinterpret(Float32, u)
end

N = 10^6
ps = Vector{Particle}(undef, N)
v = Vector{Tuple{Float32,Float32}}(undef, N)
@btime sum($energy, $ps)
@btime sum($first, $v)

```

```julia
  886.809 μs (1 allocation: 16 bytes)
  878.668 μs (1 allocation: 16 bytes)

```

---

<div class="post-metadata">

**Author:** ![zgornel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zgornel/32/217487_2.png) [@zgornel](https://discourse.julialang.org/u/zgornel)\
**Post date:** [February 12, 2019, 1:09pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/12 "2019-02-12T13:09:55Z")

</div>

I am a bit confused. Is an IAEA record always 5 bytes long ? If the record size is fixed to any number of bytes, it can be memory mapped to a `Matrix` parametrized by a custom defined `primitive type`. If the record size varies, you can still memory map it to a `BitArray` and index into it. This is the only way one can have a easily addressable/indexable representation of out-of-core objects unless you handcraft your own.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [February 12, 2019, 2:30pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/13 "2019-02-12T14:30:07Z")

</div>

I recommend against using `primitive type` for anything at all, cf eg [https://github.com/JuliaLang/julia/issues/29193](https://github.com/JuliaLang/julia/issues/29193), [https://github.com/JuliaLang/julia/issues/29053](https://github.com/JuliaLang/julia/issues/29053), [https://github.com/JuliaLang/julia/issues/26026](https://github.com/JuliaLang/julia/issues/26026), [https://discourse.julialang.org/t/odd-byte-length-primitive-types-and-reinterpret/9025/3](https://discourse.julialang.org/t/odd-byte-length-primitive-types-and-reinterpret/9025/3).

---

<div class="post-metadata">

**Author:** ![zgornel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zgornel/32/217487_2.png) [@zgornel](https://discourse.julialang.org/u/zgornel)\
**Post date:** [February 12, 2019, 2:58pm UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/14 "2019-02-12T14:58:58Z")

</div>

I was not aware of that, looks ugly indeed. Then, the regular `struct` will have to do.

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [February 13, 2019, 6:50am UTC](https://discourse.julialang.org/t/mmap-and-memory-alignment/20665/16 "2019-02-13T06:50:29Z")

</div>

Yes, the record length varies, but not within a single file. Roughly the situation is this: The IAEA files come in pairs. There is one human readable .IAEAheader and a binary .IAEAphsp file. The header describes, which particle properties are recorded in the .IAEAphsp file. For instance it is quite common, that all particles have the same `z` position. Usually in this case to save memory `z` is only stored once in the header instead of repeating it again and again for each particle.