What's the latest and greatest in data in Julia

as pointed out above, parquet + duckdb work which means you could use TidierDB.jl to work on the file directly without reading it into memory

 using TidierDB
 db = connect(duckdb())

 @chain db_table(db, "path_to_mwe2.parquet") begin
    ### your work here
    @collect #to bring it to a local df
 end
3 Likes