Reading and writing Apache arrow files

General Usage
, ,
4

Even better: We can convert the arrow table to a Pandas dataframe:

import pandas as pd
import pyarrow as pa

print("Reading arrow file...")
mmap = pa.memory_map('../data/sim_log_uncompressed.arrow')

with mmap as source:
    array = pa.ipc.open_file(source).read_all()

# print(array.schema)

# print(array[0]) # time, works
# print(array[1]) # orientation, works

# the following works, but you need one table per column
t_time   = pa.Table.from_arrays([array[0]], names=["time"])
t_orient = pa.Table.from_arrays([array[1]], names=["orient"])

# this gives just one table; 
table = array.to_pandas()
print(table)

The type of columns that contain vectors is “object”, they contain numpy arrays… Should be easy to work with…

1 Like