# Customize serialization on fetch from worker

**URL:** <https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637>\
**Category:** General Usage\
**Created:** [June 25, 2019, 4:05am UTC](https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637 "2019-06-25T04:05:42Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![felipenoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/felipenoris/32/553_2.png) [@felipenoris](https://discourse.julialang.org/u/felipenoris)\
**Post date:** [June 25, 2019, 4:05am UTC](https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637/1 "2019-06-25T04:05:42Z")

</div>

I’m working on Mongoc.jl and I would like to make BSON documents serializable between julia workers.

```julia
mutable struct BSON
    handle::Ptr{Cvoid}
end

```

Since `Mongoc.BSON` is just a wrapper for a C handle, if I just fetch the BSON from a worker, it will just copy the handle pointer address and point to an invalid location on the master process.

So I created a wrapper around BSON that converts it to a buffer as a Vector{UInt8}.

```julia
struct BufferedBSON
    bson_data::Vector{UInt8}
end

function BufferedBSON(bson::BSON)
    io = IOBuffer()
    write_bson(io, bson)
    return BufferedBSON(take!(io))
end

BSON(buff::BufferedBSON) :: BSON = read_bson(buff.bson_data)[1]

```

So that the following code works just fine:

```julia
addprocs(1)
@everywhere using Mongoc

let
    f = @spawn Mongoc.BufferedBSON(Mongoc.BSON("a" => 1))
    bson = Mongoc.BSON(fetch(f))
    @test bson["a"] == 1
end

```

But, I would like to add a syntax sugar so that the user uses only BSON, as in:

```julia
f = @spawn Mongoc.BSON("a" => 1)
bson = fetch(f)

```

Is there a way to customize the serialization primitives to hook `Mongoc.BufferedBSON` as a transport for serializing BSON in this setup?

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [June 25, 2019, 5:58am UTC](https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637/2 "2019-06-25T05:58:26Z")

</div>

I think you want to provide a definition for `Serialization.serialize(s::AbstractSerializer, b::BSON)`, and similar for `Serialization.deserialize()`?

---

<div class="post-metadata">

**Author:** ![felipenoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/felipenoris/32/553_2.png) [@felipenoris](https://discourse.julialang.org/u/felipenoris)\
**Post date:** [June 25, 2019, 2:55pm UTC](https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637/3 "2019-06-25T14:55:14Z")

</div>

By looking at `Serialization` code, looks like it cannot be extended. It also looks like the solution is to implement `read(io, bson)`, `write(io, bson)` ?

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [June 26, 2019, 1:03am UTC](https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637/4 "2019-06-26T01:03:46Z")

</div>

It is possible, but appears to be under documented and currently requires dipping into the implementation details a bit. In particular, you need to be aware of how the type information is written, and do the same in your own `serialize` function.

Here’s a demo of how to do it:

```julia
# Simplest possible example of a struct managing externally allocated data
mutable struct MyType
    p::Ptr{Int}
end

function MyType(i::Integer)
    p = reinterpret(Ptr{Int}, ccall(:malloc, Ptr{Cvoid}, (Csize_t,), sizeof(Int)))
    unsafe_store!(p, i)
    t = MyType(p)
    finalizer((m)->ccall(:free, Cvoid, (Ptr{Cvoid},), m.p), t)
    t
end

```

Now the serialization parts

```julia
using Serialization

function Serialization.serialize(s::AbstractSerializer, m::MyType)
    Serialization.serialize_type(s, typeof(m)) # Implementation detail of Serialization module.
    # Next we are free to serialize arbitrary binary data
    write(s.io, unsafe_load(m.p))
end

function Serialization.deserialize(s::AbstractSerializer, ::Type{MyType})
    # Read binary data from `s`
    val = read(s.io, Int)
    MyType(val)
end

```

As a test:

```julia
m = MyType(1234)

buf = IOBuffer()
serialize(buf, m)
seek(buf,0)

m2 = deserialize(buf)

@show m m2
@show unsafe_load(m.p) unsafe_load(m2.p)

```

which shows that `m` and `m2` now hold the same content but at different addresses.

```julia
m = MyType(Ptr{Int64} @0x000056331a1915a0)
m2 = MyType(Ptr{Int64} @0x00005633196ee530)
unsafe_load(m.p) = 1234
unsafe_load(m2.p) = 1234

```

---

<div class="post-metadata">

**Author:** ![felipenoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/felipenoris/32/553_2.png) [@felipenoris](https://discourse.julialang.org/u/felipenoris)\
**Post date:** [June 26, 2019, 1:14am UTC](https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637/5 "2019-06-26T01:14:43Z")

</div>

Thanks @c42f! I just figured it out.

The complete solution was this:

```julia
struct BufferedBSON
    bson_data::Vector{UInt8}
end

function BufferedBSON(bson::BSON)
    io = IOBuffer()
    write_bson(io, bson)
    return BufferedBSON(take!(io))
end

BSON(buff::BufferedBSON) :: BSON = read_bson(buff.bson_data)[1]

function Serialization.serialize(s::AbstractSerializer, bson::BSON)
    Serialization.serialize_type(s, BSON)
    Serialization.serialize(s.io, BufferedBSON(bson))
end

function Serialization.deserialize(s::AbstractSerializer, ::Type{BSON})
    BSON(Serialization.deserialize(s.io))
end

Base.write(io::IO, bson::BSON) = serialize(io, bson)
Base.read(io::IO, ::Type{BSON}) = deserialize(io)::BSON

```

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [June 26, 2019, 1:52am UTC](https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637/6 "2019-06-26T01:52:47Z")

</div>

```julia
function Serialization.serialize(s::AbstractSerializer, bson::BSON)
    Serialization.serialize_type(s, BSON)
    Serialization.serialize(s.io, BufferedBSON(bson))
end

```

This works but you’re writing more bytes than necessary. It’s more efficient to just use binary IO after having written the type info:

```julia
function Serialization.serialize(s::AbstractSerializer, bson::BSON)
    Serialization.serialize_type(s, BSON)
    write_bson(s.io, bson)
end

```

```julia
function Serialization.deserialize(s::AbstractSerializer, ::Type{BSON})
    read_bson(s.io) # I assume you have a read_bson returning BSON?
end

```

This way you also don’t need the BufferedBson type at all.

---

<div class="post-metadata">

**Author:** ![felipenoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/felipenoris/32/553_2.png) [@felipenoris](https://discourse.julialang.org/u/felipenoris)\
**Post date:** [June 26, 2019, 2:01am UTC](https://discourse.julialang.org/t/customize-serialization-on-fetch-from-worker/25637/7 "2019-06-26T02:01:19Z")

</div>

Yes, I also noticed this. I’ll refactor the code to do just that. Thanks a lot!