# How to dispatch this serializer function?

**URL:** <https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981>\
**Category:** Performance\
**Tags:** dispatch\
**Created:** [August 7, 2021, 6:17am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981 "2021-08-07T06:17:12Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 6:17am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/1 "2021-08-07T06:17:12Z")

</div>

```julia
"""
io = serialize(v) reinterprets a Julia object into a series of bytes.

An exercise in multiple dispatch style.

Based on
https://de.mathworks.com/matlabcentral/fileexchange/29457-serialize-deserialize
and "julianized" with the experts on
https://discourse.julialang.org/

"""

# Encode object types
type2byte(::Float64) = UInt8(0)
type2byte(::Float32) = UInt8(1)
type2byte(::Float16) = UInt8(2)
type2byte(::Bool) = UInt8(3)
type2byte(::Char) = UInt8(4)
type2byte(::String) = UInt8(5)
type2byte(::Int8) = UInt8(6)
type2byte(::UInt8) = UInt8(7)
type2byte(::Int16) = UInt8(8)
type2byte(::UInt16) = UInt8(9)
type2byte(::Int32) = UInt8(10)
type2byte(::UInt32) = UInt8(11)
type2byte(::Int64) = UInt8(12)
type2byte(::UInt64) = UInt8(13)
type2byte(::Tuple) = UInt8(100)
type2byte(::Any) = UInt8(255) # struct
type2byte(::AbstractArray) = UInt8(255) # Array of struct

# Type and size prefix
function prefix(io, x)                       
    write(io, type2byte(x))
    write(io, UInt8(0))
end
function prefix(io, x::Union{String, Tuple})
    write(io, type2byte(x))
    write(io, UInt8(1))
    write(io, UInt32(length(x)))
end
function prefix(io, x::AbstractArray)
    write(io, type2byte(first(x)))
    write(io, UInt8(ndims(x)))
    write(io, UInt32.(size(x))...)
end

# Writable (homogeneous) object types
Writable = Union{<:Real, Char, String, 
                 AbstractArray{<:Real}, AbstractArray{Char}, AbstractArray{String}}

# Entry function
function serialize(v)
    io = IOBuffer(UInt8[]; append = true)
    _serialize(io, v)
    return take!(io)
end

# Write prefix and data
function _serialize(io, v::T) where {T <: Writable}
    println("(data)\n$v\n(\\data)")
    prefix(io, v)
    write(io, v)
end
# Dispatch for Tuples or Array of Any
function _serialize(io, v::T) where {T <: Union{Tuple, AbstractArray{Any}}}
    println("(Array of Any / Tuple)")
    prefix(io, v)
    _serialize.(Ref(io), v);
end
# Dispatch for Array of Struct (stored as Struct of Array)
function _serialize(io, v::AbstractArray)
    println("(Array of Struct)")
    prefix(io, v)
    for name in fieldnames(typeof(first(v)))
        println("$(name)")
        _serialize(io, getfield.(v, name));
    end
end
# Dispatch for Struct
function _serialize(io, v)
    println("(Struct)")
    prefix(io, v)
    for name in fieldnames(typeof(v))
        println("$(name)")
        _serialize(io, getfield(v, name));
    end
end

#############################################################

# Test data
Array_of_Int = [1 ,2]
Array_of_Tuple = [(1, 2), (2, 3)]
Array_of_Any = ["Ab", (1, 2)]

mutable struct Coords
    x::Float64
    y::Float64
    z::Float64
end
Coords() = Coords(rand(), rand(), rand())
Array_of_Struct = [Coords() for i in 1:5]
Single_Struct = Coords()

# Test
println("Test Array_of_Int")
Int64.(serialize(Array_of_Int))
println()
println("Test Array_of_Tuple")
Int64.(serialize(Array_of_Tuple))
println()
println("Test Array_of_Any")
Int64.(serialize(Array_of_Any))
println()
println("Test Single_Struct")
Int64.(serialize(Single_Struct))
println()
println("Test Array_of_Struct")
Int64.(serialize(Array_of_Struct))

#@code_warntype serialize(Array_of_Any)

```

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [August 7, 2021, 7:06am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/2 "2021-08-07T07:06:37Z")

</div>

Here would be my start. (there might be some typos, but I think you’ll get the gist).

````julia
# Encode number types
classToByte(::Float64) = Int8(0)
classToByte(::Union{Float32, Float16}) = Int8(1)
classToByte(::Bool) = Int8(2)
classToByte(::Char) = Int8(3)
classToByte(::Int8) = Int8(4)
classToByte(::UInt8) = Int8(5)
classToByte(::Int16) = Int8(6)
classToByte(::UInt16) = Int8(7)
classToByte(::Int32) = Int8(8)
classToByte(::UInt32) = Int8(9)
classToByte(::Int64) = Int8(10)
classToByte(::UInt64) = Int8(11)
classToByte(x) = error("Unknown number type: $cls")

function serializej2m(v)
    
    ```
    io = serializej2m(v)
    reinterprets a Julia object into a Matlab object's series of bytes.
    
    Based on
    https://de.mathworks.com/matlabcentral/fileexchange/29457-serialize-deserialize
    
    ```
    io = IOBuffer(UInt8[]; append = true)
    serializej2m(v,io)
    return take!(io)
end

function serializej2m(io, v::T) where {T<:Real}
    # Data type.
    write(io, classToByte(T))
    # Number of dimensions.
    write(io, UInt8(2))
    # Dimensions.
    write(io, UInt32(1))
    write(io, UInt32(1))
    # Data.
    write(io, v)
end

function serializej2m(io, v::Union{Char,String})
    # Data type.
    write(io, classToByte(Char)
    # Number of dimensions.
    write(io, UInt8(2))
    # Dimensions.
    write(io, UInt32(1))
    write(io, UInt32(length(v)))
    # Data.
    write(io, v)
end
# Matlab matrix. Numeric (n-dim) or character (vector).
function serializej2m(io, v::AbstractArray{T<:Real})
    iss = false
    # Data type.
    write(io, classToByte(eltype(v)))
    # Number of dimensions.
    nd = max(2, ndims(v))
    write(io, UInt8(nd))
    # Dimensions.
    for ii = 1:nd
        write(io, UInt32(size(v, ii)))
    end
    # Data.
    write(io, v)
end

# Matlab cell array. Any content not fitting into a Matlab matrix.
function serializej2m(io, v::AbstractArray)
    # Data type.
    write(io, UInt8(254)) # 254 = cell.
    # Number of dimensions.
    nd = max(2, ndims(v))
    write(io, UInt8(nd))
    # Dimensions.
    for ii = 1:nd
        write(io, UInt32.(size(v)))
    end
    # Just serialize each member.
    for ii = 1:numel(v)
        write(io, serializej2m(v[ii]))
    end
end

````

---

<div class="post-metadata">

**Author:** ![gustaphe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gustaphe/32/18174_2.png) [@gustaphe](https://discourse.julialang.org/u/gustaphe)\
**Post date:** [August 7, 2021, 7:11am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/3 "2021-08-07T07:11:23Z")

</div>

Basically, any time you’re saying

```julia
if typeof(x) == Y
    #choose which function to run
end

```

, you mean dispatch.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [August 7, 2021, 7:19am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/4 "2021-08-07T07:19:41Z")

</div>

This one

```julia
    # Just serialize each member.
    for ii = 1:numel(v)
        write(io, serializej2m(v[ii]))
    end

```

can be slow. Probably it’s better to expand it with union splitting.

---

<div class="post-metadata">

**Author:** ![gustaphe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gustaphe/32/18174_2.png) [@gustaphe](https://discourse.julialang.org/u/gustaphe)\
**Post date:** [August 7, 2021, 7:32am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/5 "2021-08-07T07:32:25Z")

</div>

Yes, there’s two bad things on those lines. Unnecessary linear indexing and unnecessarily making a `String` layover.

```julia
serializej2m.(Ref(io), v)

```

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 7:50am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/6 "2021-08-07T07:50:55Z")

</div>

Whoa, thank you!  
Does the ordering of the dispatches matter? Or is the ordering from specific (::type) to general (cls)?  
Not heard of in the Manual or text books.

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 7:52am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/7 "2021-08-07T07:52:30Z")

</div>

Only found Tim’s [explanation](https://julialang.org/blog/2018/08/union-splitting/), but could you elaborate for this case?

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 7:54am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/8 "2021-08-07T07:54:37Z")

</div>

OK for the element-wise operator. But what is the Ref needed for?

---

<div class="post-metadata">

**Author:** ![gustaphe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gustaphe/32/18174_2.png) [@gustaphe](https://discourse.julialang.org/u/gustaphe)\
**Post date:** [August 7, 2021, 8:00am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/9 "2021-08-07T08:00:57Z")

</div>

[It chooses its methods from specific to general.](https://docs.julialang.org/en/v1/manual/methods/)

`Ref` is like a container for a value, that (in this instance) tells Julia you _don’t_ want to broadcast across it.

`println.(stdout)` gives an error, (`no method matching length(::Base.TTY)`). Essentially, you can’t broadcast across an `IO`. But you _can_ broadcast across a `Ref{IO}`, and the result is to always return that one `IO`. So

```julia
serializej2m.(io, v) # errors, because you can't take "every element of io"
serializej2m.(Ref(io), v) # works, takes every element of v and uses it with io

```

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 8:03am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/10 "2021-08-07T08:03:49Z")

</div>

Thx, clear now.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [August 7, 2021, 8:04am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/11 "2021-08-07T08:04:51Z")

</div>

> [@Bardo](#):
>
> Whoa, thank you!  
> Does the ordering of the dispatches matter? Or is the ordering from specific (::type) to general (cls)?  
> Not heard of in the Manual or text books.

The order does not matter. Julia will always pick the most specific method, no matter the order they were defined in (assuming no method is overwritten by another, that is).

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [August 7, 2021, 9:38am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/12 "2021-08-07T09:38:52Z")

</div>

Lots of repetitions in the code. I think this covers Numbers, Characters and Strings, and arrays thereof.

It doesn’t do Arrays of `Any`, because I have to leave the house. There probably needs to be some special handling of that, an extra `_serialize` method. But there would be virtually no code repetition, I’ll wager.

Warning: _Totally_ untested:

```julia
# Encode number types
classToByte(::Float64) = Int8(0)
classToByte(::Union{Float32, Float16}) = Int8(1)
classToByte(::Bool) = Int8(2)
classToByte(::Union{Char, String}) = Int8(3) 
classToByte(::Int8) = Int8(4)
classToByte(::UInt8) = Int8(5)
classToByte(::Int16) = Int8(6)
classToByte(::UInt16) = Int8(7)
classToByte(::Int32) = Int8(8)
classToByte(::UInt32) = Int8(9)
classToByte(::Int64) = Int8(10)
classToByte(::UInt64) = Int8(11)
classToByte(x) = UInt8(254)

dimensions(x) = (UInt8(2), UInt32(1), UInt32(length(x))) # works for numbers, chars and strings
dimensions(x::AbstractArray) = (UInt8(max(2, ndims(x))), UInt32.(size(x))...)

prefix(v) = (classToByte(v), dimensions(v)...)

function serializej2m(v)
    io = IOBuffer(UInt8[]; append = true)
    _serialize(io, v)
    return take!(io)
end

function _serialize(io, v)
    # Meta information 
    write(io, prefix(v)...)
    # Data
    write(io, v)
end

```

My code may be buggy or even wrong, but I think this can be expressed _extremely_ concisely with multiple dispatch, using functions like `prefix` etc.

But frankly, I think this could have been expressed almost as concisely in Matlab, too. The original code was just not well organized.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [August 7, 2021, 9:59am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/13 "2021-08-07T09:59:48Z")

</div>

Actually, I think it would be cleaner to let the serializer report the correct dimensionality, e.g. zero for scalars, instead of `max(2,ndims)`. Then the _de_serializer could take care of adjusting it if that language cannot represent scalars.

Now, the serializer is throwing away information needlessly, and misrepresenting the data, and even storing it less efficiently.

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 10:23am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/14 "2021-08-07T10:23:04Z")

</div>

Right.

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 10:26am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/15 "2021-08-07T10:26:40Z")

</div>

A lot to learn from, thx!

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 11:17am UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/16 "2021-08-07T11:17:16Z")

</div>

Remaining from @Oscar_Smith’s code, how could you dispatch this?

`if isstructtype(typeof(v))`

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [August 7, 2021, 12:36pm UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/17 "2021-08-07T12:36:43Z")

</div>

you cant. (Although `isstructtype` probably doesn’t do what you think it does and probably isn’t the right thing to use)

---

<div class="post-metadata">

**Author:** ![gustaphe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gustaphe/32/18174_2.png) [@gustaphe](https://discourse.julialang.org/u/gustaphe)\
**Post date:** [August 7, 2021, 12:46pm UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/18 "2021-08-07T12:46:43Z")

</div>

You could put `applicable(propertynames, x)` in the fallback function. Or assume that you catch everything that is not a composite type, so that the fallback simply uses `propertynames`.

Edit: all of my guesses were wrong, but the principle holds.

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 1:27pm UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/19 "2021-08-07T13:27:47Z")

</div>

Indeed, I have some doubts:  
`isstructtype(typeof(v)) # gives true for v=[1], issue?`

---

<div class="post-metadata">

**Author:** ![Bardo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bardo/32/21601_2.png) [@Bardo](https://discourse.julialang.org/u/Bardo)\
**Post date:** [August 7, 2021, 1:30pm UTC](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981/20 "2021-08-07T13:30:10Z")

</div>

good to know _applicable(),_ thx.

[Next page](https://discourse.julialang.org/t/how-to-dispatch-this-serializer-function/65981.md?page=2)