Macro for binary data deserialization into tuples of variable types

General Usage
,
1

Can somebody help with this case?

## I've started with reinterpreting some complex types from a binary stream:
bytevec = UInt8[0x01, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00]
reinterpret(Tuple{Int8, Int16, Int32}, bytevec)

# but seems like because of structure alignment I cannot just read 
# any combination of bitstypes from "packed" binary data
sizeof(bytevec) != sizeof(Tuple{Int8, Int16, Int32}) # 7 != 8

# What I want is to read structured data from a binary stream into a tuple:
io = IOBuffer(bytevec)
result = (read(io, Int8), read(io, Int16), read(io, Int32))

#1 The problem is, I cannot hardcode types, because there are variable combinations of types, known at runtime. So I need to generate a code that gathers several reads with appropriate types into a tuple. So, I’m writing a macro for that:

macro read_schema(io, Ts...)
  args = map(T -> :(read($io, $T)), Ts)
  :(tuple($(args...)))
end

# this works as expected:
bytevec = UInt8[0x01, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00]
io = IOBuffer(bytevec)
@read_schema(io, Int8, Int16, Int32)

# But then I need to write type info inside a tuple variable.
# And for some reason tuple desctructuring is not working properly with macro:
types = (Int8, Int16, Int32)
io = IOBuffer(bytevec)
@read_schema(io, types...)

ERROR: MethodError: no method matching read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{Int8}, ::Type{Int16}, ::Type{Int32})

#2 Another approach I tried with generated function, taken StructArrays as example:

@generated function read_schema_gen(io::IOBuffer, types::Type...)
    Expr(:tuple, [Expr(:call, :read, :io, t) for t in types]...)
end

bytevec = UInt8[0x01, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00]
io = IOBuffer(bytevec)
types = (Int8, Int16, Int32)
read_schema_gen(io, types...)

ERROR: The IO stream does not support reading objects of type Type{Int8}.

I don’t understand, what’s wrong with approaches #1 and #2, and how to fix that?

2

Why isn’t

io = IOBuffer(bytevec)
result = (read(io, Int8), read(io, Int16), read(io, Int32))

sufficient? I feel I’m missing something from you problem description, but it sounds like a very familiar use case to parse tuples dependent on runtime data. Macros are source code transformations, so i don’t see how they could help you in this case.

If all you want is to have a function as your read_schema:

julia> readschema(v, types...) = (io=IOBuffer(v);tuple((read(io,x) for x in types)...))
readschema (generic function with 1 method)

julia> bytevec = UInt8[0x01, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00]
7-element Array{UInt8,1}:
 0x01
 0x02
 0x00
 0x03
 0x00
 0x00
 0x00

julia> readschema(bytevec, Int8, Int16, Int32)
(1, 2, 3)

julia> typeof(ans)
Tuple{Int8,Int16,Int32}

I’m sure you can figure that this function isn’t production ready but it demonstrates a point.

3

Maybe I didn’t describe the problem well, but there are variable combinations of types. Assume user sending requests for some specific data fields, each time a new combination of known types. And then receiving them in binary format. So, I cannot explicitly write type literals, like result = (read(io, Int8), read(io, Int16), read(io, Int32)), but I have a tuple of concrete combination of types.

Thank you, this is a very simple and nice solution! Don’t know, why I started overcomplicating it with macro stuff. I rewrote it as follows: (since there can be an array of such tuples to read from a larger binary stream)

function readschema(io::IOBuffer, types::Type...)
    tuple((read(io,x) for x in types)...)
end

It is type-stable and fast for repeated calls? Is that loop over static tuple unwrapped into several reads at compilation?

However, I have spent some time with that macro and generated functions stuff, so I will leave this question opened, until those ploblems are clear to me.

4

Looks like this code has some overhead comparing to explicit call:

function readschema(io::IOBuffer, types::Type...)
    tuple((read(io,x) for x in types)...)
end
function readschema_static(io::IOBuffer)
    (read(io, Int8), read(io, Int16), read(io, Int32))
end

types = (Int8, Int16, Int32)
io = IOBuffer(rand(UInt8, 100))
@time readschema(io, types...) # 0.000035 seconds (17 allocations: 592 bytes)
@time readschema_static(io) # 0.000001 seconds
5

what about

function readschema2(io::IOBuffer, types)
    map(type -> read(io,type), types)
end

For me, it times as 3.105 ns (using @btime)

6

It is much faster than the first solution, but still much slower than static schema - testing for large arrays:

readschema_static(io::IOBuffer) = (read(io, Int8), read(io, Int16), read(io, Int32))
function readvec_static(data::Vector{UInt8}, types::Type...)
    elsize = mapreduce(sizeof, +, types)
    len = sizeof(data) ÷ elsize
    result = Vector{Tuple{types...}}(undef, len)
    io = IOBuffer(data)
    @inbounds for i in 1:len
        result[i] = readschema_static(io)
    end
    result
end

readschema(io::IOBuffer, types::Type...) = tuple((read(io,x) for x in types)...)
function readvec(data::Vector{UInt8}, types::Type...)
    elsize = mapreduce(sizeof, +, types)
    len = sizeof(data) ÷ elsize
    result = Vector{Tuple{types...}}(undef, len)
    io = IOBuffer(data)
    @inbounds for i in 1:len
        result[i] = readschema(io, types...)
    end
    result
end

readschema2(io::IOBuffer, types) = map(type -> read(io,type), types)
function readvec2(data::Vector{UInt8}, types::Type...)
    elsize = mapreduce(sizeof, +, types)
    len = sizeof(data) ÷ elsize
    result = Vector{Tuple{types...}}(undef, len)
    io = IOBuffer(data)
    @inbounds for i in 1:len
        result[i] = readschema2(io, types)
    end
    result
end

types = (Int8, Int16, Int32)
elsize = mapreduce(sizeof, +, types)
bytevec = rand(UInt8, 1000 * elsize)

@time readvec_static(bytevec, types...)  # 0.000120 seconds (2.99 k allocations: 70.641 KiB)
@time readvec(bytevec, types...)         # 0.004136 seconds (18.97 k allocations: 632.828 KiB)
@time readvec2(bytevec, types...)        # 0.000843 seconds (5.98 k allocations: 117.359 KiB)

(You can refine exact values with @btime.)

7

I think you are just measuring compile time. By the second run, they are equal.

8

They are not equal on repeated runs.
Ok, I will run it with BenchmarkTools:

using BenchmarkTools

@btime readvec_static($bytevec, types...) # 77.000 μs (2993 allocations: 70.64 KiB)
@btime readvec($bytevec, types...)        # 3.949 ms (18973 allocations: 632.83 KiB)
@btime readvec2($bytevec, types...)       # 548.600 μs (5983 allocations: 117.36 KiB)
9

For the generated function, you’d have to do it like this:

@generated function read_schema_gen2(io::IOBuffer, types::Type...)
    reads = (:(read(io, $(t.parameters[1]))) for t in types)
    return :(tuple($(reads...)))
end

types is a tuple of Type a the time the generation happens, so you have to get out the parameters.

I think it won’t be getting better than that:

julia> @code_lowered read_schema_gen2(IOBuffer(bytevec), types...)
CodeInfo(
    @ REPL[38]:2 within `read_schema_gen2'
   ┌ @ REPL[38] within `macro expansion'
1 ─│ %1 = Main.read(io, Int8)
│  │ %2 = Main.read(io, Int16)
│  │ %3 = Main.read(io, Int32)
│  │ %4 = Main.tuple(%1, %2, %3)
└──│      return %4
   └
)
1 Like
10

Alternative solution is here: Difference between Type{T} and T when passing type variable inside generated function? - #4 by Elrod