[ANN] ProtoBuf.jl 1.0.0

drvi
Wed, 17 Aug 2022 19:14:22 +0000
Package Announcements

ProtoBuf.jl 1.0.0

At RelationalAI, we created a new Julia package for working with the Protocol Buffers format and I’m happy to announce that this complete rewrite will soon be registered as a (very) breaking version of the current ProtoBuf.jl, that is 1.0.0. This new release brings following benefits:

The generated Julia structs and codec methods often result in noticeably faster serialization with lower memory overhead.*

We dropped protoc_jll dependency, which currently carries a large (100+MB) binary. Time to first protojl (our variant of protoc) is also much lower.

Enumerations are now generated using the EnumX.jl package, meaning they are a proper subtypes of Base.Enum.

* For example, see the following benchmark from issue #179 “Deserialization is extremely slow for messages with many small sub-messages.”

Here’s the pre-1.0 ProtoBuf.jl:

julia> BenchmarkTools.@benchmark(read_example_proto()) |> display
BenchmarkTools.Trial: 71 samples with 1 evaluation.
 Range (min … max):  66.590 ms … 76.673 ms  ┊ GC (min … max): 0.00% … 10.80%
 Time  (median):     71.134 ms              ┊ GC (median):    5.37%
 Time  (mean ± σ):   70.467 ms ±  2.704 ms  ┊ GC (mean ± σ):  4.42% ±  3.48%

  █ ▁▆ ▁▁                      ▆   ▁  ▁  ▃  ▃
  █▄██▇██▄▄▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▄▄▄▇▇█▄▄▄█▇▁█▇▄█▇▇█▄▇▇▁▁▁▁▁▁▁▁▁▁▁▁▄ ▁
  66.6 ms         Histogram: frequency by time        75.8 ms <

 Memory estimate: 33.81 MiB, allocs estimate: 625475.

and here is ProtoBuf.jl 1.0.0:

julia> BenchmarkTools.@benchmark(read_example_proto_new()) |> display
BenchmarkTools.Trial: 741 samples with 1 evaluation.
 Range (min … max):  6.541 ms …   9.074 ms  ┊ GC (min … max): 0.00% … 26.07%
 Time  (median):     6.647 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   6.748 ms ± 386.710 μs  ┊ GC (mean ± σ):  1.16% ±  4.39%

  ▄▆█▇▅▃▂▁▂
  ██████████▄▅▅▇▇▆▅▁▁▁▄▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▁▄▅▄▁▅▄▄▁▄▄▆▅▄▇ ▇
  6.54 ms      Histogram: log(frequency) by time      8.65 ms <

 Memory estimate: 1.85 MiB, allocs estimate: 2668.

As mentioned earlier, this release is very breaking, migrating to 1.0.0 will require some effort. Here are some differences to the pre-1.0.0 version that you should take into account:

Services and RPCs are not yet implemented. We will focus on these in near future as a part of our effort to build native gRPC libraries for Julia.

All generated structs are immutable and don’t share a common abstract type. By default, no convenience constructors are generated for these structs, but you might use add_kwarg_contructors=true option in protojl.

oneof fields are now translated to OneOf{T} types with fields name::Symbol and val::T containing the name and the value of the chosen member.

Nested definitions (e.g. message Parent { message Child {} }) would previously be named like Parent_Child, now the Child message would be translated to a struct called var"Parent.Child".

When translating proto files with a package directive, protojl will generate a directory structure that copies the levels of said package. In the future, we want to support generation of full blown Julia packages that are easy to register in (private) registries.

Please see the docs for more information about the package.

Also please note that in this case the 1.0.0 version was not meant to signal stability of the package, there are no planned breaking changes, but there might be some rough edges here and there, so please report any issues you encounter.

20 posts - 11 participants