Trying to read parquet file that is using Zstd codec

ldsands
Tue, 24 Dec 2019 15:31:44 +0000
General Usage

Hello everyone,

I would like to read a parquet file into a Julia DataFrame. Unfortunately for my Julia skills (but fortunately for my hard drive) the codec used to compress the parquet file was zstd. I have played around with the CodecZstd and TranscodingStreams along with FileIO and I can’t seem to get it to load much less get the data into a DataFrame. Any help would be appreciated.

Below is the only code I’ve gotten to work but I have no idea what to do with this line output.

using Glob, CodecZstd, DataFrames

p = dirname(@__FILE__)
parent_dir = splitdir(p)[1]
parquet_folder = joinpath(parent_dir, "month_parquet_RS")
println(parquet_folder)
parquet_files = glob("*.parquet", parquet_folder)
parquet_files = parquet_files[1]

proc = open(parquet_files)
stream = ZstdCompressorStream(proc)
for line in eachline(stream)
    println(line)
end
close(stream)

7 posts - 2 participants