Using ProtoBuf to serialize data

New to Julia
1

Hi there,

I’m trying to use ProtoBuf to serialize data but I’m coming across an error I don’t know how to solve:

MWE:

using ProtoBuf

abstract type MyAbstractType end

struct MyType <: MyAbstractType
    ts::UInt64
    symbol::String
    price::Float32
    size::Float32
end

pb = PipeBuffer()

test = MyType(time_ns(), "Test", 100, 10)

msg = writeproto(pb, test)

The error I get is:

MethodError: no method matching meta(::Type{MyType})
Closest candidates are:
  meta(::ProtoMeta, ::Type, ::Vector{Pair{Symbol, Union{String, Type}}}, ::Vector{Symbol}, ::Vector{Int64}, ::Dict{Symbol, Any}) at /home/jcooper/.julia/packages/ProtoBuf/l7nsO/src/codec.jl:576
  meta(::ProtoMeta, ::Type, ::Vector{Pair{Symbol, Union{String, Type}}}, ::Vector{Symbol}, ::Vector{Int64}, ::Dict{Symbol, Any}, ::Vector{Symbol}) at /home/jcooper/.julia/packages/ProtoBuf/l7nsO/src/codec.jl:576
  meta(::ProtoMeta, ::Type, ::Vector{Pair{Symbol, Union{String, Type}}}, ::Vector{Symbol}, ::Vector{Int64}, ::Dict{Symbol, Any}, ::Vector{Symbol}, ::Dict) at /home/jcooper/.julia/packages/ProtoBuf/l7nsO/src/codec.jl:576
  ...

Stacktrace:
 [1] writeproto(io::IOBuffer, obj::MyType)
   @ ProtoBuf ~/.julia/packages/ProtoBuf/l7nsO/src/codec.jl:419
 [2] top-level scope
   @ In[20]:16
 [3] eval
   @ ./boot.jl:360 [inlined]
 [4] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
   @ Base ./loading.jl:1094

I don’t understand this error at all. If anyone has any experience with this package guidance would be appreciated.

As an aside is there any downside to using the Serialization package from stdlib instead if I only intend on programming in Julia?

Thanks.

2

Unless things changes recently, you need to write the specification of your structure in a .Proto file, compile it, which will generate a file defining your structure, then include it.

You cannot define directly the structures by hand, in Julia, unless you really know what you are doing

3

The Serialization stdlib isn’t designed to be an on disk data storage format. The big consequence of this is that the storage format is considered an implementation detail and therefore subject to change between julia releases. For example, if you serialize a data structure with the Serialization stdlib from a julia 1.5 session, there’s no guarantee you’ll be able to read the file from julia 1.6. I believe the main use of the built in serialization is for communication between different julia processes. That said, if you just want a quick and dirty way to save your struct so you can load it tomorrow, then Serialization might be fine.

If you’re only working in julia two options for writing user defined structs to file are the JLD and JLD2 packages.

2 Likes
4

Thanks for the help so far! I was definitely missing a few steps…

I have now:

  • Created a .proto file.
  • Generated a .jl file using ProtoBuf.protoc
  • Included that .jl file.

That gives me a ProtoType that I can use to writeproto on. However, when I read from this stream instead of just getting the values I get the entire ProtoType structure.

Reading the pipebuffer gives:

ProtoMetaAttribs[ProtoMetaAttribs(1, :ts, :uint64, UInt64, 0, false, UInt64[0x0000000000000000], nothing), ProtoMetaAttribs(2, :symbol, :string, AbstractString, 0, false, [""], nothing), ProtoMetaAttribs(3, :bidPx, :float, Float32, 0, false, Float32[0.0], nothing), ProtoMetaAttribs(4, :askPx, :float, Float32, 0, false, Float32[0.0], nothing), ProtoMetaAttribs(5, :bidSz, :float, Float32, 0, false, Float32[0.0], nothing), ProtoMetaAttribs(6, :askSz, :float, Float32, 0, false, Float32[0.0], nothing), ProtoMetaAttribs(7, :exchgId, :int64, Int64, 0, false, [0], nothing)]
, Dict{Symbol, Any}(:symbol => "Test", :exchgId => 1, :bidPx => 1.0f0, :ts => 0x00010a1470638c57, :askPx => 1.0f0, :bidSz => 1.0f0, :askSz => 1.0f0), Set{Symbol}())

In the ProtoBuf.jl documentation it seems to indicate I can create a subtype of ProtoType. I tried doing this but since ProtoType is a concrete type I’m not allowed.

Should my ProtoType be a concrete type? (Possibly I’ve done an earlier step incorrectly). If so how do I create a struct through which I can read/write just the values instead of the entire structure.

5

If you must use proto for interoperability with legacy systems, then you must use proto.

As an aside is there any downside to using the Serialization package from stdlib instead if I only intend on programming in Julia?

From this, it sounds like you are in more of a greenfield situation. In this case, I very strongly advise against touching protobuf with a 10-foot pole. Someone really needs to write the definite “Protobuf: A fractal of bad design” article. They have good marketing, though: “Protobuf – Building the legacy of tomorrow, today”.

Quoting the official proto documentation (Techniques | Protocol Buffers Documentation):

Protocol Buffers are not designed to handle large messages. As a general rule of thumb, if you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy.

I am in the unfortunate situation at work of having to slowly migrate parts of a nontrivial java/scala/golang/python/C++/node.js product away from proto. Please, if you can avoid proto, then do so.

edit: If you need to deal with business data, consider json / xml + lz4. (lz4 removes most of the verbosity). If you need to deal with large heterogenous binary data, consider the lovely blobs.jl + mmap. If you need to deal with large homogeneous arrays of primitives, consider hdf5. Afaiu there are no good julia cap’n’proto bindings, but that is also a nice wire format.

2 Likes
6

Correct, but this may circumvent that limitation (haven’t tried it yet though):