Unpacking binary data into a Julia struct

General Usage
,
1

I’m having a hard time finding an example in Julia v1.0+ for handling unpacking binary data into a Julia struct. Is there an easy way to do this?

Say I have some custom struct Test and wanted to read some binary data from stream io, is there someway to fill all the fields of a test struct very quickly?

julia> struct Test
           a::UInt16
           b::UInt16
           c::UInt32
       end

julia> read(io, Test)
ERROR: The IO stream does not support reading objects of type Test.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] read(::IOStream, ::Type) at ./io.jl:917
 [3] top-level scope at REPL[40]:1

julia> Test([read(io, t) for t in fieldtypes(Test)]...) #something that isn't this....
Test(0x002a, 0x0008, 0x2a270010)
1 Like
2

You need to define how to read the data from the stream.

function Base.read(io::IO, ::Type{Test})
    Test(read(io,UInt16),read(io,UInt16),read(io,UInt32))
end

As an example, this is how Complex numbers are handled in Base.

function read(s::IO, ::Type{Complex{T}}) where T<:Real
    r = read(s,T)
    i = read(s,T)
    Complex{T}(r,i)
end
4 Likes
3

That’s a pretty elegant solution that slipped my mind. I guess I thought there was a way without implementing my own function for reading

4

I would suggest to consider the native Serialization solution. See

using Serialization: serialize, deserialize

struct Test
   a::UInt16
   b::UInt16
   c::UInt32
end

test = Test(0, 0, 0)

# write to io
open("plsdel.io", "w") do io
   serialize(io, test)
end

# read from io
test_deserialized = open("plsdel.io", "r") do io
   deserialize(io)
end

test == test_deserialized # true

This is less manual and likely to be well-support as Serialization is in base.

5

Super clever solution. I’ve been reading in binary data in ad-hocish ways.

One approach:
Iterate over the field names and types in the struct. Yes this works, I just forget the function to get field names and types from structs right now :smiley: . So defining the struct, defines the types, and its all stored sequentially.

It falls apart a bit with Chars/strings, buuuut you can patch that with a dict if you know the length.

6

Interestingly, loading into an Array of Test with read! will work:

A = Array{Test}(undef, 1)
read!(io, A)

Somehow read! knows how to read from the stream, but read doesn’t. Is it intended to be this way?

1 Like
7

Bumping this question–can I rely on read!(f, Vector{MyStruct}(undef, 1))[1], or might there be unexpected sharp edges? The documentation is pretty terse:

help?> read!

  read!(stream::IO, array::AbstractArray)
  read!(filename::AbstractString, array::AbstractArray)

  Read binary data from an I/O stream or file, filling in array.

…alignment? padding?

8

I think that you get C-style padding and alignment. Cf

https://github.com/pao/StrPack.jl

and similar packages for alternantives.