# Macro for binary data deserialization into tuples of variable types

**URL:** <https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088>\
**Category:** General Usage\
**Tags:** question, binaryio\
**Created:** [April 23, 2020, 1:47pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088 "2020-04-23T13:47:21Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [April 23, 2020, 1:47pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/1 "2020-04-23T13:47:21Z")

</div>

Can somebody help with this case?

```julia
## I've started with reinterpreting some complex types from a binary stream:
bytevec = UInt8[0x01, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00]
reinterpret(Tuple{Int8, Int16, Int32}, bytevec)

# but seems like because of structure alignment I cannot just read 
# any combination of bitstypes from "packed" binary data
sizeof(bytevec) != sizeof(Tuple{Int8, Int16, Int32}) # 7 != 8

# What I want is to read structured data from a binary stream into a tuple:
io = IOBuffer(bytevec)
result = (read(io, Int8), read(io, Int16), read(io, Int32))

```

* * *

#1 The problem is, I cannot hardcode types, because there are variable combinations of types, known at runtime. So I need to generate a code that gathers several reads with appropriate types into a tuple. So, I’m writing a macro for that:

```julia
macro read_schema(io, Ts...)
  args = map(T -> :(read($io, $T)), Ts)
  :(tuple($(args...)))
end

# this works as expected:
bytevec = UInt8[0x01, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00]
io = IOBuffer(bytevec)
@read_schema(io, Int8, Int16, Int32)

# But then I need to write type info inside a tuple variable.
# And for some reason tuple desctructuring is not working properly with macro:
types = (Int8, Int16, Int32)
io = IOBuffer(bytevec)
@read_schema(io, types...)

```

`ERROR: MethodError: no method matching read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{Int8}, ::Type{Int16}, ::Type{Int32})`

* * *

#2 Another approach I tried with generated function, taken [StructArrays](https://github.com/JuliaArrays/StructArrays.jl/blob/master/src/interface.jl) as example:

```julia
@generated function read_schema_gen(io::IOBuffer, types::Type...)
    Expr(:tuple, [Expr(:call, :read, :io, t) for t in types]...)
end

bytevec = UInt8[0x01, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00]
io = IOBuffer(bytevec)
types = (Int8, Int16, Int32)
read_schema_gen(io, types...)

```

`ERROR: The IO stream does not support reading objects of type Type{Int8}.`

I don’t understand, what’s wrong with approaches #1 and #2, and how to fix that?

---

<div class="post-metadata">

**Author:** ![laborg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laborg/32/5474_2.png) [@laborg](https://discourse.julialang.org/u/laborg)\
**Post date:** [April 23, 2020, 4:13pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/2 "2020-04-23T16:13:52Z")

</div>

Why isn’t

```julia
io = IOBuffer(bytevec)
result = (read(io, Int8), read(io, Int16), read(io, Int32))

```

sufficient? I feel I’m missing something from you problem description, but it sounds like a very familiar use case to parse tuples dependent on runtime data. Macros are source code transformations, so i don’t see how they could help you in this case.

If all you want is to have a function as your `read_schema`:

```julia
julia> readschema(v, types...) = (io=IOBuffer(v);tuple((read(io,x) for x in types)...))
readschema (generic function with 1 method)

julia> bytevec = UInt8[0x01, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00]
7-element Array{UInt8,1}:
 0x01
 0x02
 0x00
 0x03
 0x00
 0x00
 0x00

julia> readschema(bytevec, Int8, Int16, Int32)
(1, 2, 3)

julia> typeof(ans)
Tuple{Int8,Int16,Int32}

```

I’m sure you can figure that this function isn’t production ready but it demonstrates a point.

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [April 23, 2020, 5:17pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/3 "2020-04-23T17:17:14Z")

</div>

> [@laborg](#):
>
> Why isn’t
> 
> ```julia
> io = IOBuffer(bytevec)
> result = (read(io, Int8), read(io, Int16), read(io, Int32))
> 
> ```
> 
> sufficient?

Maybe I didn’t describe the problem well, but there are variable combinations of types. Assume user sending requests for some specific data fields, each time a new combination of known types. And then receiving them in binary format. So, I cannot explicitly write type literals, like `result = (read(io, Int8), read(io, Int16), read(io, Int32))`, but I have a tuple of concrete combination of types.

> [@laborg](#):
>
> readschema(v, types…) = (io=IOBuffer(v);tuple((read(io,x) for x in types)…))

Thank you, this is a very simple and nice solution! Don’t know, why I started overcomplicating it with macro stuff. I rewrote it as follows: (since there can be an array of such tuples to read from a larger binary stream)

```julia
function readschema(io::IOBuffer, types::Type...)
    tuple((read(io,x) for x in types)...)
end

```

It is type-stable and fast for repeated calls? Is that loop over static tuple unwrapped into several reads at compilation?

* * *

However, I have spent some time with that macro and generated functions stuff, so I will leave this question opened, until those ploblems are clear to me.

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [April 23, 2020, 5:26pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/4 "2020-04-23T17:26:48Z")

</div>

> [@sairus7](#):
>
> It is type-stable and fast for repeated calls? Is that loop over static tuple unwrapped into several reads at compilation?

Looks like this code has some overhead comparing to explicit call:

```julia
function readschema(io::IOBuffer, types::Type...)
    tuple((read(io,x) for x in types)...)
end
function readschema_static(io::IOBuffer)
    (read(io, Int8), read(io, Int16), read(io, Int32))
end

types = (Int8, Int16, Int32)
io = IOBuffer(rand(UInt8, 100))
@time readschema(io, types...) # 0.000035 seconds (17 allocations: 592 bytes)
@time readschema_static(io) # 0.000001 seconds

```

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [April 23, 2020, 5:51pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/5 "2020-04-23T17:51:11Z")

</div>

what about

```julia
function readschema2(io::IOBuffer, types)
    map(type -> read(io,type), types)
end

```

For me, it times as 3.105 ns (using `@btime`)

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [April 23, 2020, 7:12pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/6 "2020-04-23T19:12:19Z")

</div>

It is much faster than the first solution, but still much slower than static schema - testing for large arrays:

```julia
readschema_static(io::IOBuffer) = (read(io, Int8), read(io, Int16), read(io, Int32))
function readvec_static(data::Vector{UInt8}, types::Type...)
    elsize = mapreduce(sizeof, +, types)
    len = sizeof(data) ÷ elsize
    result = Vector{Tuple{types...}}(undef, len)
    io = IOBuffer(data)
    @inbounds for i in 1:len
        result[i] = readschema_static(io)
    end
    result
end

readschema(io::IOBuffer, types::Type...) = tuple((read(io,x) for x in types)...)
function readvec(data::Vector{UInt8}, types::Type...)
    elsize = mapreduce(sizeof, +, types)
    len = sizeof(data) ÷ elsize
    result = Vector{Tuple{types...}}(undef, len)
    io = IOBuffer(data)
    @inbounds for i in 1:len
        result[i] = readschema(io, types...)
    end
    result
end

readschema2(io::IOBuffer, types) = map(type -> read(io,type), types)
function readvec2(data::Vector{UInt8}, types::Type...)
    elsize = mapreduce(sizeof, +, types)
    len = sizeof(data) ÷ elsize
    result = Vector{Tuple{types...}}(undef, len)
    io = IOBuffer(data)
    @inbounds for i in 1:len
        result[i] = readschema2(io, types)
    end
    result
end

types = (Int8, Int16, Int32)
elsize = mapreduce(sizeof, +, types)
bytevec = rand(UInt8, 1000 * elsize)

@time readvec_static(bytevec, types...) # 0.000120 seconds (2.99 k allocations: 70.641 KiB)
@time readvec(bytevec, types...) # 0.004136 seconds (18.97 k allocations: 632.828 KiB)
@time readvec2(bytevec, types...) # 0.000843 seconds (5.98 k allocations: 117.359 KiB)

```

(You can refine exact values with `@btime`.)

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [April 23, 2020, 7:19pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/7 "2020-04-23T19:19:59Z")

</div>

I think you are just measuring compile time. By the second run, they are equal.

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [April 23, 2020, 7:26pm UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/8 "2020-04-23T19:26:57Z")

</div>

They are not equal on repeated runs.  
Ok, I will run it with BenchmarkTools:

```julia
using BenchmarkTools

@btime readvec_static($bytevec, types...) # 77.000 μs (2993 allocations: 70.64 KiB)
@btime readvec($bytevec, types...) # 3.949 ms (18973 allocations: 632.83 KiB)
@btime readvec2($bytevec, types...) # 548.600 μs (5983 allocations: 117.36 KiB)

```

---

<div class="post-metadata">

**Author:** ![phg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phg/32/17184_2.png) [@phg](https://discourse.julialang.org/u/phg)\
**Post date:** [April 24, 2020, 10:15am UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/9 "2020-04-24T10:15:26Z")

</div>

For the generated function, you’d have to do it like this:

```julia
@generated function read_schema_gen2(io::IOBuffer, types::Type...)
    reads = (:(read(io, $(t.parameters[1]))) for t in types)
    return :(tuple($(reads...)))
end

```

`types` is a tuple of `Type` a the time the generation happens, so you have to get out the parameters.

I think it won’t be getting better than that:

```julia
julia> @code_lowered read_schema_gen2(IOBuffer(bytevec), types...)
CodeInfo(
    @ REPL[38]:2 within `read_schema_gen2'
   ┌ @ REPL[38] within `macro expansion'
1 ─│ %1 = Main.read(io, Int8)
│ │ %2 = Main.read(io, Int16)
│ │ %3 = Main.read(io, Int32)
│ │ %4 = Main.tuple(%1, %2, %3)
└──│ return %4
   └
)

```

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [April 24, 2020, 10:34am UTC](https://discourse.julialang.org/t/macro-for-binary-data-deserialization-into-tuples-of-variable-types/38088/10 "2020-04-24T10:34:08Z")

</div>

Alternative solution is here: [Difference between Type{T} and T when passing type variable inside generated function? - #4 by Elrod](https://discourse.julialang.org/t/difference-between-type-t-and-t-when-passing-type-variable-inside-generated-function/38121/4)