# Trying to read parquet file that is using Zstd codec

**URL:** https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657
**Category:** General Usage
**Tags:** question, dataframes, zstd, parquet
**Created:** [December 24, 2019, 3:31pm UTC](https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657 "2019-12-24T15:31:44Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![ldsands](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ldsands/32/2800_2.png) [@ldsands](https://discourse.julialang.org/u/ldsands)
#### Post date: [December 24, 2019, 3:31pm UTC](https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657/1 "2019-12-24T15:31:44Z")

</div>

Hello everyone,

I would like to read a parquet file into a Julia DataFrame. Unfortunately for my Julia skills (but fortunately for my hard drive) the codec used to compress the parquet file was zstd. I have played around with the CodecZstd and TranscodingStreams along with FileIO and I can’t seem to get it to load much less get the data into a DataFrame. Any help would be appreciated.

Below is the only code I’ve gotten to work but I have no idea what to do with this line output.

```nohighlight
using Glob, CodecZstd, DataFrames

p = dirname(@ __FILE__ )
parent_dir = splitdir(p)[1]
parquet_folder = joinpath(parent_dir, "month_parquet_RS")
println(parquet_folder)
parquet_files = glob("*.parquet", parquet_folder)
parquet_files = parquet_files[1]

proc = open(parquet_files)
stream = ZstdCompressorStream(proc)
for line in eachline(stream)
    println(line)
end
close(stream)

```

---

<div class="post-metadata">

### Author: ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 24, 2019, 9:22pm UTC](https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657/2 "2019-12-24T21:22:23Z")

</div>

Try Parquet Files.jl

---

<div class="post-metadata">

### Author: ![ldsands](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ldsands/32/2800_2.png) [@ldsands](https://discourse.julialang.org/u/ldsands)
#### Post date: [December 24, 2019, 9:32pm UTC](https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657/3 "2019-12-24T21:32:23Z")

</div>

This is the result of using ParquetFiles.jl (also this is based upon Parquet.jl)

```nohighlight
ERROR: LoadError: Unknown compression codec for column chunk: 6

```

Nither FileIO nor ParquetFiles supports zstd at the moment. I think I need to use CodecZstd but I’m not sure how.

---

<div class="post-metadata">

### Author: ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 24, 2019, 9:42pm UTC](https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657/4 "2019-12-24T21:42:03Z")

</div>

Parquet support for julia is wip

---

<div class="post-metadata">

### Author: ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 24, 2019, 9:51pm UTC](https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657/5 "2019-12-24T21:51:39Z")

</div>

your best bet is to PyCall or RCall to read the parquet, save it as feather and the load it into Julia

---

<div class="post-metadata">

### Author: ![ldsands](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ldsands/32/2800_2.png) [@ldsands](https://discourse.julialang.org/u/ldsands)
#### Post date: [December 26, 2019, 9:33pm UTC](https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657/6 "2019-12-26T21:33:25Z")

</div>

I made a [pull request](https://github.com/JuliaIO/Parquet.jl/pull/41) that might add support for zstd compressed parquet files. I’ll post here if the pull request gets approved. I’m not sure it will though I have no formal computer science training and I’ve never submitted a pull request before so I’ve probably not done something correct.

---

<div class="post-metadata">

### Author: ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [May 6, 2020, 5:45am UTC](https://discourse.julialang.org/t/trying-to-read-parquet-file-that-is-using-zstd-codec/32657/7 "2020-05-06T05:45:26Z")

</div>

@ldsands

Hey, Zstd is working in [Diban.jl](https://github.com/xiaodaigh/Diban.jl). I am working (slowly) to try and get this back into Parquet.jl.

But it was a simple addition. If you can’t wait just try Diban.jl

```julia
]activate diban-test
]dev Thrift

# add a particular version of Parquet.jl with fixes
]add https://github.com/xiaodaigh/Parquet.jl#zj/fix-reader

# add the latest version of Dìbǎn
]add https://github.com/xiaodaigh/Diban.jl

```

And read your ZSTD compressed file using

```julia
read_parquet(path_to_file)

```
