Use protobuf to generate jl code


#1

Hi all

I am using the ProtoBuf Julia package to generate jl from .proto file. The package is https://github.com/JuliaIO/ProtoBuf.jl/blob/master/PROTOC.md

I follow the guiding that use the command:
protoc -I=/proto-folder --julia_out=/the-folder/that/you/want/to/save/the/jl-file something.proto
and I already append the path of plugin into the bash_profile.

Here is the error:
–julia_out: protoc-gen-julia: Plugin failed with status code 127.

I check some similar question

It seems like the plugin is failed so it could not use the --Julia_out to generate jl file?

Any advices is helpful. Thanks.


#2

Have you tried to use the --plugin flag?

protoc --grpc_out=/tmp --plugin=protoc-gen-grpc=grpc_cpp_plugin foo.proto

As per this comment:


#3

Looks like protoc is not able to load something, either the plugin or a dependent shared library. Can you double check if the addition to PATH is effective and if protoc and Julia are working fine? Also try --plugin as @SalchiPapa suggests.


#4

Yes, I tried this command and if I use
protoc --plugin=protoc-gen-julia=plugin --julia_out=proto test/proto/plugin.proto
the error is:
descriptor.proto: File not found.
test/proto/plugin.proto: Import “descriptor.proto” was not found or had errors.
test/proto/plugin.proto:76:12: “FileDescriptorProto” is not defined.
the descriptor.proto is located in the proto folder. So I think might be the location is missing
So I try
protoc --plugin=protoc-gen-julia=plugin --julia_out=proto test/proto/plugin.proto -I=test/proto/plugin.proto
The error is :
[libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax specified for the proto file: . Please use ‘syntax = “proto2”;’ or ‘syntax = “proto3”;’ to specify a syntax version. (Defaulted to proto2 syntax.)
plugin: program not found or is not executable
–julia_out: protoc-gen-julia: Plugin failed with status code 1.

I double checked the path and the path could been seen when I run echo $PATH in a new terminal.

So confused about this


#5

I use the echo $PATH in a new terminal to check the path, the path seems updated. I also try the protoc --plugin=protoc-gen-NAME=path/to/mybinary --NAME_out=OUT_DIR
from
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.compiler.plugin
by
protoc --plugin=protoc-gen-julia=plugin --julia_out=proto test/proto/plugin.proto

but it keep saying can not find the file that located in the proto folder. So I add the -I flag to give the path, but the error just the same as pervious.
(protoc --plugin=protoc-gen-julia=plugin --julia_out=proto test/proto/plugin.proto -I=test/proto
plugin: program not found or is not executable
–julia_out: protoc-gen-julia: Plugin failed with status code 1.
)


#6

Please note that:

  • folder proto must be present already
  • when specifying --plugin=protoc-gen-julia=... the value should be full path to plugin executable.
  • when giving relative paths, make sure they are appropriate from where you are running the command

For example:

protoc -I=test/proto --plugin=protoc-gen-julia=path_to_dot_julia/v0.6/ProtoBuf/plugin/protoc-gen-julia
–julia_out=proto test/proto/plugin.proto

Replace path_to_dot_julia with the path to the place where julia packages are kept (usually $HOME/.julia)


#7

Thanks for the notes.
I checked the content in the .Julia but I am confused about how to use this file. Do you mean just replace the /User/Name/.Julia into the path of the proto-gen-julia? I double checked the path before executing my command but still not work
" ============================================================================
" Netrw Directory Listing (netrw v156)
" /Users/Name/.julia
" Sorted by name
" Sort sequence: [/],\<core\%(\.\d\+\)\=\>,\.h,.c$,.cpp$,~=*,*,\.o,
" Quick Help: :help -:go up dir D:delete R:rename s:sort-by x:special
" ==============================================================================
…/
./
.cache/
lib/
v0.6/


#8

.julia is not a file, it is a directory
the subdirectory v0.6 is the place where packages are kept
look in that directory and you will see a number of Packages given as there own directory names.

I have no experience with protobuf, so one of the others will pick this up with you.


#9

For what it’s worth, in a package where I’m using ProtoBuf, I have the following utility function to run the ProtoBuf compiler from Julia, in Linux. (Package renamed to Foo to protect innocent company secrets.)

function generate_protobuf_layer(proto_dir; remove_existing_files = false)
    # Find all *.proto files.
    proto_files = filter(x -> endswith(x, ".proto"), readdir(proto_dir))
    # Add the path to each file.
    proto_files = [joinpath(proto_dir, f) for f in proto_files]

    # Add the directory of the Julia plugin for Protobuf to the path so
    # the Protobuf compiler can find it.
    plugin_dir = joinpath(Pkg.dir("ProtoBuf"), "plugin")
    ENV["PATH"] = string(plugin_dir, ":", ENV["PATH"])

    # Add the path to the Julia binary so the Protobuf compiler can
    # call back to Julia.
    ENV["PATH"] = string(JULIA_HOME, ":", ENV["PATH"])
    
    # This instructs the plugin to generate a Foo_pb module instead of a
    # Foo module.
    ENV["JULIA_PROTOBUF_MODULE_POSTFIX"] = 1

    # Generate the files directly into the src/Foo_pb directory.
    outdir = joinpath(Pkg.dir("Foo"), "src", "Foo_pb")
    if remove_existing_files
        for f in readdir(outdir)
            rm(joinpath(outdir, f))
        end
    end

    # Call the Protobuf compiler.
    run(`protoc -I=$(proto_dir) --julia_out=$(outdir) $(proto_files)`)
end

#10

Following the steps outlined in Generating Code (from .proto files) I was able to compile the relevant code for the proto/plugin.proto example. There is one small extra step that is missing which prevents you from executing the example verbatim.

Consider the following steps.

  1. Is protoc in your PATH? Type protoc --help. You should see the following:
Usage: protoc [OPTION] PROTO_FILES
Parse PROTO_FILES and generate output based on the options given:
  -IPATH, --proto_path=PATH   Specify the directory in which to search for
                              imports.  May be specified multiple times;
                              directories will be searched in order.  If not
                              given, the current working directory is used.
  --version                   Show version info and exit.
  -h, --help                  Show this text and exit.
  --encode=MESSAGE_TYPE       Read a text-format message of the given type
                              from standard input and write it in binary
                              to standard output.  The message type must
                              be defined in PROTO_FILES or their imports.
  --decode=MESSAGE_TYPE       Read a binary message of the given type from
                              standard input and write it in text format
                              to standard output.  The message type must
                              be defined in PROTO_FILES or their imports.
  --decode_raw                Read an arbitrary protocol message from
                              standard input and write the raw tag/value
                              pairs in text format to standard output.  No
                              PROTO_FILES should be given when using this
                              flag.
  -oFILE,                     Writes a FileDescriptorSet (a protocol buffer,
    --descriptor_set_out=FILE defined in descriptor.proto) containing all of
                              the input files to FILE.
  --include_imports           When using --descriptor_set_out, also include
                              all dependencies of the input files in the
                              set, so that the set is self-contained.
  --include_source_info       When using --descriptor_set_out, do not strip
                              SourceCodeInfo from the FileDescriptorProto.
                              This results in vastly larger descriptors that
                              include information about the original
                              location of each decl in the source file as
                              well as surrounding comments.
  --error_format=FORMAT       Set the format in which to print errors.
                              FORMAT may be 'gcc' (the default) or 'msvs'
                              (Microsoft Visual Studio format).
  --print_free_field_numbers  Print the free field numbers of the messages
                              defined in the given proto files. Groups share
                              the same field number space with the parent 
                              message. Extension ranges are counted as 
                              occupied fields numbers.
  --plugin=EXECUTABLE         Specifies a plugin executable to use.
                              Normally, protoc searches the PATH for
                              plugins, but you may specify additional
                              executables not in the path using this flag.
                              Additionally, EXECUTABLE may be of the form
                              NAME=PATH, in which case the given plugin name
                              is mapped to the given executable even if
                              the executable's own name differs.
  --cpp_out=OUT_DIR           Generate C++ header and source.
  --java_out=OUT_DIR          Generate Java source file.
  --python_out=OUT_DIR        Generate Python source file.

  1. Navigate to the /.julia/v0.6/ProtoBuf/plugin directory and add it to your PATH environment variable. For example, you can type pwd in the directory which in my case returns: /home/zygmunt/.julia/v0.6/ProtoBuf/plugin. To add the directory to your path for the current terminal session you can type the following commands in your terminal (with your actual path of course)
       PATH=$PATH:/home/zygmunt/.julia/v0.6/ProtoBuf/plugin
       export PATH

Verify that the plugin folder has been added to the PATH by typing echo $PATH in the terminal.

  1. Navigate to the /.julia/v0.6/ProtoBuf/test directory.

  2. Create the jlout directory by typing mkdir jlout. This explicit step is missing in the aforementioned tutorial.

  3. Type protoc -I=proto --julia_out=jlout proto/plugin.proto. This should generate 3 files in the jlout directory: descriptor_pb.jl; google.jl and plugin_pb.jl. Based on the output of protoc --help you can also type protoc --proto_path=proto --julia_out=jlout proto/plugin.proto.

Try these steps and post again if it still does not work.


#11

So far, your comment is the most detailed help for generating code from .proto files in julia which I found so far, thank you for that.
Do you know, if there is also apossibility to use this package to generate the code on a Windows Machine?


#12

I think it should be possible to use it on a Windows machine as well with some small modification. It won’t work on Windows at the moment because https://github.com/JuliaIO/ProtoBuf.jl/blob/master/plugin/protoc-gen-julia is a shell script. I suspect all that is needed is to write an equivalent Windows batch script. I don’t have access to a Windows machine at the moment, but can probably try it out next week.


#13

Thanks for your answer!
I have already spoken to tanmaykm and as he told me the same, I have written a very basic batch script to replace protoc-gen-julia with my new file protoc-gen-julia.bat, this worked for my case. At the moment, it doesn’t work for special cases where FLAGS aren’t set, I try to fix this and upload it as soon as I am finished.

However, if you know your way around batch scripts, I am glad to receive some help or get inputs to improve my implementation.

My Implementation so far is quite simple:

Add in the same folder where the protoc-gen-julia file is located (in github: https://github.com/JuliaIO/ProtoBuf.jl/blob/master/plugin/protoc-gen-julia) a filed named
protoc-gen-julia.bat with the content:

@echo off

julia  -e "using ProtoBuf; using ProtoBuf.Gen; gen()"

and run for example the following command:

protoc -I=proto --plugin=protoc-gen-julia=<PATH>\protoc-gen-julia.bat --julia_out=jlout proto/plugin.proto

EDIT: The functionality for generating julia code from .proto files on Windows Machines in now integrated in the GitHub repo.