Read gzipped protobuf file

I have a “protobuf reader”, generated using protoc, proto/coordinate_push.jl, using which I’d like to import a gzipped file. This file contains a list of frames, each containing object coordinates.

The corresponding Python code, which I try to migrate, is:

frameList = []

with gzip.open(filePath) as f:
	data = f.read()
	next_pos, pos = 0, 0

	while pos < len(data):
		msg = coordinate_push.CoordinatesFrame()

		next_pos, pos = _DecodeVarint32(data, pos)
		msg.ParseFromString(data[pos:pos + next_pos])

		frameList.append(msg)

		pos += next_pos

Being new to Julia and generally not strong at IO (also in Python), I’m unsure how to to start. The following code yields an ERROR: LoadError: UndefVarError: load not defined.

 include("proto/coordinate_push.jl")
 using FileIO
 using ProtoBuf
 file_path = "sensor_1502063868355_1502064000053.dat.gz" #example
 obj = load(file_path) #I have also tried explicitly FileIO.load()
 readproto(obj, CoordinatesFrame())

In some more detail, the error reads as

 [1] handle_error(::UndefVarError, ::File{DataFormat{:GZIP}}) at C:\Users\unis\.juliapro\JuliaPro_v1.4.2-1\packages\FileIO\zL0JE\src\error_handling.jl:82
 [2] handle_exceptions(::Array{Any,1}, ::String) at C:\Users\unis\.juliapro\JuliaPro_v1.4.2-1\packages\FileIO\zL0JE\src\error_handling.jl:77

I had initially asked this question on StackOverflow. I’m using Julia 1.4.2 (which I could update if needed) on Windows (latest version).

My guess is that somehow FileIO.load() can’t deal with gzipped files. I would highly appreciate any comments or hints that help migrating the above Python code (including also the replication of the while loop, if possible). Thank you!

For the gzip reading part you want to look into https://github.com/JuliaIO/CodecZlib.jl.

1 Like