# \[ANN\] ProtoBuf.jl 1.0.0

**URL:** <https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885>\
**Category:** Package Announcements\
**Tags:** package, protobuf\
**Created:** [August 17, 2022, 7:14pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885 "2022-08-17T19:14:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![drvi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drvi/32/13236_2.png) [@drvi](https://discourse.julialang.org/u/drvi)\
**Post date:** [August 17, 2022, 7:14pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/1 "2022-08-17T19:14:22Z")

</div>

## ProtoBuf.jl 1.0.0

At RelationalAI, we created a new Julia package for working with the Protocol Buffers format and I’m happy to announce that this complete rewrite will soon be [registered](https://github.com/JuliaRegistries/General/pull/66434) as a (very) breaking version of the current [`ProtoBuf.jl`](https://github.com/JuliaIO/ProtoBuf.jl), that is `1.0.0`. This new release brings following benefits:

- The generated Julia structs and codec methods often result in noticeably faster serialization with lower memory overhead.\*
- We dropped `protoc_jll` dependency, which currently carries a large (100+MB) binary. Time to first `protojl` (our variant of `protoc`) is also much lower.
- Enumerations are now generated using the `EnumX.jl` package, meaning they are a proper subtypes of `Base.Enum`.

\* For example, see the following benchmark from [issue #179](https://github.com/JuliaIO/ProtoBuf.jl/issues/179) _“Deserialization is extremely slow for messages with many small sub-messages.”_

Here’s the pre-1.0 `ProtoBuf.jl`:

```julia-auto
julia> BenchmarkTools.@benchmark(read_example_proto()) |> display
BenchmarkTools.Trial: 71 samples with 1 evaluation.
 Range (min … max): 66.590 ms … 76.673 ms ┊ GC (min … max): 0.00% … 10.80%
 Time (median): 71.134 ms ┊ GC (median): 5.37%
 Time (mean ± σ): 70.467 ms ± 2.704 ms ┊ GC (mean ± σ): 4.42% ± 3.48%

  █ ▁▆ ▁▁ ▆ ▁ ▁ ▃ ▃
  █▄██▇██▄▄▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▄▄▄▇▇█▄▄▄█▇▁█▇▄█▇▇█▄▇▇▁▁▁▁▁▁▁▁▁▁▁▁▄ ▁
  66.6 ms Histogram: frequency by time 75.8 ms <

 Memory estimate: 33.81 MiB, allocs estimate: 625475.

```

and here is `ProtoBuf.jl 1.0.0`:

```julia-auto
julia> BenchmarkTools.@benchmark(read_example_proto_new()) |> display
BenchmarkTools.Trial: 741 samples with 1 evaluation.
 Range (min … max): 6.541 ms … 9.074 ms ┊ GC (min … max): 0.00% … 26.07%
 Time (median): 6.647 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 6.748 ms ± 386.710 μs ┊ GC (mean ± σ): 1.16% ± 4.39%

  ▄▆█▇▅▃▂▁▂
  ██████████▄▅▅▇▇▆▅▁▁▁▄▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▁▄▅▄▁▅▄▄▁▄▄▆▅▄▇ ▇
  6.54 ms Histogram: log(frequency) by time 8.65 ms <

 Memory estimate: 1.85 MiB, allocs estimate: 2668.

```

As mentioned earlier, this release is very breaking, migrating to `1.0.0` will require some effort. Here are some differences to the pre-1.0.0 version that you should take into account:

- **`Service`s and `RPC`s are not yet implemented**. We will focus on these in near future as a part of our effort to build native gRPC libraries for Julia.
- All generated structs are immutable and don’t share a common abstract type. By default, no convenience constructors are generated for these structs, but you might use `add_kwarg_contructors=true` option in `protojl`.
- `oneof` fields are now translated to `OneOf{T}` types with fields `name::Symbol` and `val::T` containing the name and the value of the chosen member.
- Nested definitions (e.g. `message Parent { message Child {} }`) would previously be named like `Parent_Child`, now the `Child` message would be translated to a struct called `var"Parent.Child"`.
- When translating proto files with a `package` directive, `protojl` will generate a directory structure that copies the levels of said `package`. In the future, we want to support generation of full blown Julia packages that are easy to register in (private) registries.

Please see the [docs](https://juliaio.github.io/ProtoBuf.jl/dev/) for more information about the package.

Also please note that in this case the 1.0.0 version was not meant to signal stability of the package, there are no planned breaking changes, but there might be some rough edges here and there, so please report any issues you encounter.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [August 18, 2022, 12:26am UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/2 "2022-08-18T00:26:57Z")

</div>

Any particular reason to not call the package `ProtocolBuffers.jl`?

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [August 18, 2022, 12:45am UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/3 "2022-08-18T00:45:09Z")

</div>

Huh, I just noticed that the package name is spelled `ProtoBuf`, not `ProtoBuff`. If you’re going to abbreviate `Protocol Buffer`, I would have expected it to be spelled `ProtoBuff`.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [August 18, 2022, 12:58am UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/4 "2022-08-18T00:58:20Z")

</div>

protobuf is how it’s usually shortened: [GitHub - protocolbuffers/protobuf: Protocol Buffers - Google's data interchange format](https://github.com/protocolbuffers/protobuf)

---

<div class="post-metadata">

**Author:** ![drvi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drvi/32/13236_2.png) [@drvi](https://discourse.julialang.org/u/drvi)\
**Post date:** [August 18, 2022, 5:32am UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/5 "2022-08-18T05:32:25Z")

</div>

FWIW, during its development, the package was called [ProtocolBuffers.jl](https://github.com/Drvi/ProtocolBuffers.jl), but since `ProtoBuf.jl` was already established, we chose not to fragment the ecosystem with a competing implementation.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [August 18, 2022, 3:14pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/6 "2022-08-18T15:14:37Z")

</div>

> [@drvi](#):
>
> The generated Julia structs and codec methods often result in noticeably faster serialization with lower memory overhead.

How does it compare to the reference C++ implementation?

---

<div class="post-metadata">

**Author:** ![viralbshah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viralbshah/32/54_2.png) [@viralbshah](https://discourse.julialang.org/u/viralbshah)\
**Post date:** [August 18, 2022, 4:50pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/7 "2022-08-18T16:50:53Z")

</div>

Would it make sense to merge this into the existing ProtoBuf.jl package as v2.0?

---

<div class="post-metadata">

**Author:** ![drvi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drvi/32/13236_2.png) [@drvi](https://discourse.julialang.org/u/drvi)\
**Post date:** [August 18, 2022, 7:18pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/8 "2022-08-18T19:18:21Z")

</div>

> How does it compare to the reference C++ implementation?

Not sure! The google repo does contain some benchmarking code I was able to run today, but I’m not sure if I can easily make an apples to apples comparison with our code, the setup they use seemed quite involved… I’ll look into it again at some point.

> Would it make sense to merge this into the existing ProtoBuf.jl package as v2.0?

We bumped the existing ProtoBuf.jl package from 0.11.5 → 1.0.0, sorry if that was not clear!  
Edit: Assuming you brought this up for semver/compat reasons, this version change is as breaking as a bump to 2.0 would be.

---

<div class="post-metadata">

**Author:** ![senhalil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/senhalil/32/42473_2.png) [@senhalil](https://discourse.julialang.org/u/senhalil)\
**Post date:** [November 7, 2022, 4:00pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/9 "2022-11-07T16:00:08Z")

</div>

Do you plan to add a json parser/writer? There is a related [git issue](https://github.com/JuliaIO/ProtoBuf.jl/issues/66) but not much information there (or under the linked issues).

---

<div class="post-metadata">

**Author:** ![arhik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arhik/32/8991_2.png) [@arhik](https://discourse.julialang.org/u/arhik)\
**Post date:** [November 8, 2022, 7:37pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/10 "2022-11-08T19:37:47Z")

</div>

Hello, Thanks for working on this package. I am a user of this package and I noticed that encoding of unpacked representations is withheld (commented out here → [Allow unpacked repeated primitives](https://github.com/JuliaIO/ProtoBuf.jl/commit/eb6459be8ceac6ae8aa577c6105667c649cc9556)). Is there any reason its not included yet. I can see it needs test set. Any other reasons ?

---

<div class="post-metadata">

**Author:** ![drvi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drvi/32/13236_2.png) [@drvi](https://discourse.julialang.org/u/drvi)\
**Post date:** [November 8, 2022, 9:06pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/11 "2022-11-08T21:06:32Z")

</div>

At some point yes, but it’s not a priority, as one can use `protoc_jll` as a workaround, see e.g. [Parsing ProtoBuf Text Format - #3 by FireCrumb](https://discourse.julialang.org/t/parsing-protobuf-text-format/84721/3)

---

<div class="post-metadata">

**Author:** ![drvi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drvi/32/13236_2.png) [@drvi](https://discourse.julialang.org/u/drvi)\
**Post date:** [November 8, 2022, 9:09pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/12 "2022-11-08T21:09:55Z")

</div>

This would be better to discuss in an issue, but the reason is that I don’t understand the use case – it is strictly more efficient to encode the array as packed. Maybe I’m missing something.

---

<div class="post-metadata">

**Author:** ![arhik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arhik/32/8991_2.png) [@arhik](https://discourse.julialang.org/u/arhik)\
**Post date:** [November 9, 2022, 2:27am UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/13 "2022-11-09T02:27:34Z")

</div>

When supporting an existing protocol which uses unpacked representation for encoding and decoding we need it. If it is strictly encoding and decoding at the other end it is not important. But before decoding if hash is used to verify the object received in remote session then we would get into issues. The hash of the encoded representation will be different and will be rejected in our use case. For now I had to make a fork with these changes and continue using it. It would be ideal if changes are in the upstream. I was assuming you guys must have a reason and didn’t bother to raise an issue. I hope you are convinced about the use-case. If I am the protocol designer and implementer I would definitely avoid unpacked representation but to adhere to existing protocol it would make sense to support it.

raised a pull request here [Allow unpacked repeated primitives by arhik · Pull Request #224 · JuliaIO/ProtoBuf.jl (github.com)](https://github.com/JuliaIO/ProtoBuf.jl/pull/224)

---

<div class="post-metadata">

**Author:** ![FireCrumb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/firecrumb/32/35370_2.png) [@FireCrumb](https://discourse.julialang.org/u/FireCrumb)\
**Post date:** [November 9, 2022, 9:00am UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/14 "2022-11-09T09:00:59Z")

</div>

> [@drvi](#):
>
> At some point yes, but it’s not a priority, as one can use `protoc_jll` as a workaround, see e.g. [Parsing ProtoBuf Text Format - #3 by FireCrumb](https://discourse.julialang.org/t/parsing-protobuf-text-format/84721/3)

Just to note that I’ve updated the code in the post above to ProtoBuf 1.0.0, and am using it – in case someone wants to discuss/ask/collaborate.

It does require booth **protoc** binary, and **protoc\_jil** and **.proto** files all to be accessible in runtime.

---

<div class="post-metadata">

**Author:** ![senhalil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/senhalil/32/42473_2.png) [@senhalil](https://discourse.julialang.org/u/senhalil)\
**Post date:** [November 9, 2022, 11:55am UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/15 "2022-11-09T11:55:02Z")

</div>

Thanks for the pointer and thanks for the update @FireCrumb.

Maybe I am missing something but I would like to parse a JSON file into proto and not a protobuf text file. We are using json as the human readable intermediary and several systems are already dumping dependent on it, I checked the protobuf text format but it is quite different then json.

Do you have a suggestion for reading/parsing a json file (or a Dictionary object) into a ProtoType object directly or via the binary encoding in Julia?

---

<div class="post-metadata">

**Author:** ![drvi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drvi/32/13236_2.png) [@drvi](https://discourse.julialang.org/u/drvi)\
**Post date:** [November 9, 2022, 12:09pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/16 "2022-11-09T12:09:54Z")

</div>

Ah, there is not automatic mapping between JSON and ProtoBuf (that I know of), the closest to that was a recent proposal to integrate tightly with `StructTypes.jl`. In the end we decided that this would be best implemented in a separate package.

---

<div class="post-metadata">

**Author:** ![senhalil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/senhalil/32/42473_2.png) [@senhalil](https://discourse.julialang.org/u/senhalil)\
**Post date:** [November 9, 2022, 12:43pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/17 "2022-11-09T12:43:38Z")

</div>

Thanks @drvi ! Yeah, I was mislead by other protobuf APIs having JSON parser/writers and c++ api referring to JSON as “proto3 JSON format” but they are all custom parsers I guess.

---

<div class="post-metadata">

**Author:** ![FireCrumb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/firecrumb/32/35370_2.png) [@FireCrumb](https://discourse.julialang.org/u/FireCrumb)\
**Post date:** [November 9, 2022, 6:52pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/18 "2022-11-09T18:52:53Z")

</div>

> [@senhalil](#):
>
> Maybe I am missing something but I would like to parse a JSON file into proto and not a protobuf text file. We are using json as the human readable intermediary and several systems are already dumping dependent on it, I checked the protobuf text format but it is quite different then json.

Useful references I could find

TL;DR: unrelated to Julia, functionality exists in C++ API but not in protoc, some people implemented their cli tools with this functionality

- [decode to json from command line #4024](https://github.com/protocolbuffers/protobuf/issues/4024)
- [Protobuf JSON Converter](https://github.com/hq6/ProtobufJson)
- [Convert between JSON and Protobuf](https://www.baeldung.com/java-convert-json-protobuf)
- [Can protoc or any tool do JSON-protobuf conversion?](https://stackoverflow.com/questions/62740880/can-protoc-or-any-tool-do-json-protobuf-conversion)
- [proto-convert](https://github.com/iamAzeem/proto-convert)

* * *

Edit: seems they are using [Google::Protobuf::DescriptorPool.generated\_pool.lookup](https://googleapis.dev/python/protobuf/latest/google/protobuf/descriptor_pool.html) to load [google.protobuf.json\_format](https://googleapis.dev/python/protobuf/latest/google/protobuf/json_format.html)  
Perhaps it can be done with [Julia Interop](https://github.com/JuliaInterop)?

---

<div class="post-metadata">

**Author:** ![okartal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/okartal/32/44818_2.png) [@okartal](https://discourse.julialang.org/u/okartal)\
**Post date:** [September 22, 2023, 3:41pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/19 "2023-09-22T15:41:19Z")

</div>

> [@drvi](#):
>
> **`Service`s and `RPC`s are not yet implemented**. We will focus on these in near future as a part of our effort to build native gRPC libraries for Julia.

@drvi and @quinnj, thanks for this great package but regarding gRPC, would it make sense to wrap the [grpc/grpc: The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#) (github.com)](https://github.com/grpc/grpc) library and create a jll instead of developing a native gRPC library? Is that a bad strategy, given ProtoBuf.jl v1?

I just want to understand the rationale, I have never tried to build a jll and use it in Julia.

---

<div class="post-metadata">

**Author:** ![Jose\_Ghislain\_Quenum](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jose_ghislain_quenum/32/204363_2.png) [@Jose\_Ghislain\_Quenum](https://discourse.julialang.org/u/Jose_Ghislain_Quenum)\
**Post date:** [August 19, 2025, 1:08pm UTC](https://discourse.julialang.org/t/ann-protobuf-jl-1-0-0/85885/20 "2025-08-19T13:08:58Z")

</div>

Hi all, I am experiencing an issue with the ProtoBuf package. I am writing a simple RPC call between a client and a server. The proto file looks like the following.

```proto
syntax = "proto3";
message MyReq {
  string name = 1;
}

message MyResp {
  string greeting = 1;
}

```

After generating the stubs and skeletons (or bindings) I add the client and server as follows

```julia
include("out_folder/example_pb.jl")
using Sockets
using ProtoBuf

const PORT = 8080

function handle_reqs(the_clt_socket)
        try
                # Read request from client
                the_req = decode(ProtoDecoder(the_clt_socket), example_pb.MyReq)

                the_resp_ctn = "Hi, $(the_req.name)!"
                the_resp = example_pb.MyResp(the_resp_ctn)

                pe = ProtoEncoder(the_clt_socket) 
                encode(pe, the_resp)
                flush(the_clt_sock)

        catch e
                println("an error occurred during communication...")
                error_msg = sprint(showerror, e)
                println("Error message: ", error_msg)
        finally
                close(the_clt_sock)
        end
end

function start_server()
        server_socket = listen(PORT)
        println("Server listening on port $PORT...")

        while true
                a_sock = accept(server_socket)
                @async handle_reqs(a_sock)
        end
end

```

```julia
include("out_folder/example_pb.jl")
using ProtoBuf
using Sockets

const SERVER_IP = ip"127.0.0.1"
const PORT = 8080

function submit_request(name::String)
        clt_sock = connect(SERVER_IP, PORT)
        try 
                gr_req = example_pb.MyReq(name)
                e = ProtoEncoder(clt_sock)
                encode(e, gr_req)
                flush(sock)
                
                d = ProtoDecoder(clt_sock)
                gr_resp = decode(d, example_pb.MyResp)
                
                println("Received response: $(gr_resp.greeting)")
        catch e
                println("An error occurred during communication $e")
        finally
                close(clt_sock)
        end
end

```

But, when I run the server and client, I get the error Error message: MethodError: no method matching position(::TCPSocket)  
The function `position` exists, but no method is defined for this combination of argument types.  
There seems to be an issue with the position function called within the generated bindings. Has anyone experienced that? What’s the fix?