Reading FlatBuffer from mmaped file

New to Julia
1

I want to read a custom format written from python and used as IPC format between a python backend and a Angular frontend. The IPC format is based in Arrow Stream IPC Format. I have many Arrow tables (each one is based in a table or view from the database) with a footer written using a custom Flatbuffer format. The footer says the offset and name of each Arrow Table. So, as first step, I want to read that footer from a mmaped file, but i can’t find how to do that.

The documentation in:

http://juliadata.github.io/FlatBuffers.jl/stable/

Says I have to use a function “read” to read the flatbuffer, but i cant find that funcion. A similar function exists:

readbuffer(t::Array{UInt8,1}, pos::Int64, ::Type{T})

So, the question is: is there a example to read a flatbuffers buffer from a Array{UInt8, 1} or from a mmaped file?

EDIT: Sorry, the function read exists, but i don’t viewed it because autocomplete in Jupyter is not showing it, but if I test with Flatbuffers.read, there is a function read. But, when a pass a Array{UInt8, 1} to that function, the message showed is:

MethodError: no method matching read(::Array{UInt8,1})
Closest candidates are:
  read(!Matched::Base.DevNullStream, !Matched::Type{UInt8}) at coreio.jl:12
  read(!Matched::IOStream) at iostream.jl:481
  read(!Matched::IOStream, !Matched::Type{UInt8}) at iostream.jl:393
2

flatbuffers are build according to a specific “schema”, so if you know the schema type of the flatbuffer, file, the answer here is to call FlatBuffers(FlatBufferType, bytes).

On a related note, there is the Arrow.jl repo that supports reading the in-memory arrow format (though I dont’ believe it quite handles all the IPC formats yet). In any case, that would probably be a good package to reference since it’s doing a similar thing to what you’re after for the in-memory structure.

2 Likes