Read gzipped protobuf file

General Usage
1

I have a “protobuf reader”, generated using protoc, proto/coordinate_push.jl, using which I’d like to import a gzipped file. This file contains a list of frames, each containing object coordinates.

The corresponding Python code, which I try to migrate, is:

frameList = []

with gzip.open(filePath) as f:
	data = f.read()
	next_pos, pos = 0, 0

	while pos < len(data):
		msg = coordinate_push.CoordinatesFrame()

		next_pos, pos = _DecodeVarint32(data, pos)
		msg.ParseFromString(data[pos:pos + next_pos])

		frameList.append(msg)

		pos += next_pos

Being new to Julia and generally not strong at IO (also in Python), I’m unsure how to to start. The following code yields an ERROR: LoadError: UndefVarError: load not defined.

 include("proto/coordinate_push.jl")
 using FileIO
 using ProtoBuf
 file_path = "sensor_1502063868355_1502064000053.dat.gz" #example
 obj = load(file_path) #I have also tried explicitly FileIO.load()
 readproto(obj, CoordinatesFrame())

In some more detail, the error reads as

 [1] handle_error(::UndefVarError, ::File{DataFormat{:GZIP}}) at C:\Users\unis\.juliapro\JuliaPro_v1.4.2-1\packages\FileIO\zL0JE\src\error_handling.jl:82
 [2] handle_exceptions(::Array{Any,1}, ::String) at C:\Users\unis\.juliapro\JuliaPro_v1.4.2-1\packages\FileIO\zL0JE\src\error_handling.jl:77

I had initially asked this question on StackOverflow. I’m using Julia 1.4.2 (which I could update if needed) on Windows (latest version).

My guess is that somehow FileIO.load() can’t deal with gzipped files. I would highly appreciate any comments or hints that help migrating the above Python code (including also the replication of the while loop, if possible). Thank you!

2

For the gzip reading part you want to look into https://github.com/JuliaIO/CodecZlib.jl.

1 Like