Mmap and memory alignment

General Usage
1

I have binary files (IAEA phase space files) which contain large contiguous lists of particles. And I want to mmap these files.

Here is a simplified variant of my issue: A particle consists of 5 bytes, where the first byte encodes the type UInt8 and the other bytes encode the energy Float32. Now one way to mimic this in julia would be:

struct Particle
    type::UInt8
    energy::Float32
end

However there is a problem. sizeof(Particle) = 8, while the particles on disk are only 5 bytes. This means I cannot mmap them. Are there recommended solutions to this?
My idea was to instead do:

struct Particle
    bytes::NTuple{5,UInt8}
end

and then reinterpret chunks of the bytes field. However one problem I have is that reinterpret is quite picky:

bytes = (0x0,0x0,0x0,0x0,0x0)
reinterpret(Float32, bytes[2:5])

bitcast: expected primitive type value for second argument

So is this generally a good approach? If so, how to properly extract the energy field in the above example?

2

I’m not sure what the best approach is in julia 1.0.

We were dealing with the same issue trying to mmap structs that didn’t align in LasIO. We ended up memory mapping a Vector{UInt8}, and putting that into our own AbstractVector which could be used to retrieve individual structs. This was mostly based on the approach in GitHub - JuliaArrays/UnalignedVectors.jl: Create arrays from memory buffers that lack appropriate alignment.

Only UnalignedVectors.jl package dates from 0.6, and has not been updated to 1.0. Probably this has made it easier now:

reinterpret now works on any AbstractArray using the new ReinterpretArray type.
This supersedes the old behavior of reinterpret on Arrays. As a result, reinterpreting
arrays with different alignment requirements (removed in 0.6) is once again allowed (https://github.com/JuliaLang/julia/pull/23750).

2 Likes
4

One solution is to create a primitive type Iaea40 covering the 40 bits of information and manually implementing the interface needed to reinterpret it as a Particle (with which you do the processing). So, from memory-mapped Matrix{Iaea40} the getters and setters would do the conversion to and from Particle i.e. reinterpret(Particle, Iaea40). I never tried such an approach so this is somewhat theoretical but it should work :slight_smile:

5 Likes
5

While

doesn’t work, this does:

julia> bytes2 = [0x0, 0x1, 0x2, 0x3, 0x4]
5-element Array{UInt8,1}:
 0x00
 0x01
 0x02
 0x03
 0x04

julia> reinterpret(Float32, bytes2[2:5])
1-element reinterpret(Float32, ::Array{UInt8,1}):
 1.5399896e-36

and if you are looking to avoid allocations/want your Particle type to be a bitstype, you could use an SVector instead:

julia> using StaticArrays

julia> bytes3 = @SVector [0x0, 0x1, 0x2, 0x3, 0x4]
5-element SArray{Tuple{5},UInt8,1,5}:
 0x00
 0x01
 0x02
 0x03
 0x04

julia> reinterpret(Float32, bytes3[2:5])
1-element reinterpret(Float32, ::Array{UInt8,1}):
 1.5399896e-36
1 Like
6

@zgornel This is a nice idea. Unfortunatelly in my case, the size of Particle may vary depending on some header file, so this is not really an option.

7

Unfortunately this has very bad performance:

using StaticArrays, BenchmarkTools

x = @SVector zeros(UInt8, 4)
@benchmark reinterpret($Float32, $x)

BenchmarkTools.Trial: 
  memory estimate:  32 bytes
  allocs estimate:  2
  --------------
  minimum time:     69.666 ns (0.00% GC)
  median time:      75.958 ns (0.00% GC)
  mean time:        85.552 ns (10.43% GC)
  maximum time:     59.599 μs (99.82% GC)
  --------------
  samples:          10000
  evals/sample:     974
8

The easiest option is just define your own getters:

struct Particle
    bytes::NTuple{5,UInt8}
end

particletype(p::Particle) = p.bytes[1]
function energy(p::Particle)
    u = (UInt32(p.bytes[2]) << 24) | (UInt32(p.bytes[3]) << 16) | (UInt32(p.bytes[4]) << 8) | UInt32(p.bytes[5])
    reinterpret(Float32, u)
end
1 Like
9

This shouldn’t be a problem as long as the particle size for one mmaped region stays constant, right?

10

@Sukera right, this was in response to

where the size is hard coded once and for all (I like the approach otherwise).

11

Nice! I was worried, that performance could be bad, but it is not:

using BenchmarkTools
struct Particle
    bytes::NTuple{5,UInt8}
end

particletype(p::Particle) = p.bytes[1]
function energy(p::Particle)
    u = (UInt32(p.bytes[2]) << 24) | (UInt32(p.bytes[3]) << 16) | (UInt32(p.bytes[4]) << 8) | UInt32(p.bytes[5])
    reinterpret(Float32, u)
end

N = 10^6
ps = Vector{Particle}(undef, N)
v = Vector{Tuple{Float32,Float32}}(undef, N)
@btime sum($energy, $ps)
@btime sum($first, $v)

  886.809 μs (1 allocation: 16 bytes)
  878.668 μs (1 allocation: 16 bytes)
12

I am a bit confused. Is an IAEA record always 5 bytes long ? If the record size is fixed to any number of bytes, it can be memory mapped to a Matrix parametrized by a custom defined primitive type. If the record size varies, you can still memory map it to a BitArray and index into it. This is the only way one can have a easily addressable/indexable representation of out-of-core objects unless you handcraft your own.

13

I recommend against using primitive type for anything at all, cf eg https://github.com/JuliaLang/julia/issues/29193, https://github.com/JuliaLang/julia/issues/29053, https://github.com/JuliaLang/julia/issues/26026, https://discourse.julialang.org/t/odd-byte-length-primitive-types-and-reinterpret/9025/3.

1 Like
14

I was not aware of that, looks ugly indeed. Then, the regular struct will have to do.

16

Yes, the record length varies, but not within a single file. Roughly the situation is this: The IAEA files come in pairs. There is one human readable .IAEAheader and a binary .IAEAphsp file. The header describes, which particle properties are recorded in the .IAEAphsp file. For instance it is quite common, that all particles have the same z position. Usually in this case to save memory z is only stored once in the header instead of repeating it again and again for each particle.