# Parquet2jl: ThriftInvalidFinalReadState: while attempting to read Parquet2.Metadata.Statistics

**URL:** <https://discourse.julialang.org/t/parquet2jl-thriftinvalidfinalreadstate-while-attempting-to-read-parquet2-metadata-statistics/116668>\
**Category:** Data\
**Tags:** parquet\
**Created:** [July 5, 2024, 2:43pm UTC](https://discourse.julialang.org/t/parquet2jl-thriftinvalidfinalreadstate-while-attempting-to-read-parquet2-metadata-statistics/116668 "2024-07-05T14:43:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![PavloPolovyi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavlopolovyi/32/210479_2.png) [@PavloPolovyi](https://discourse.julialang.org/u/PavloPolovyi)\
**Post date:** [July 5, 2024, 2:43pm UTC](https://discourse.julialang.org/t/parquet2jl-thriftinvalidfinalreadstate-while-attempting-to-read-parquet2-metadata-statistics/116668/1 "2024-07-05T14:43:31Z")

</div>

Got this error when trying to open parquet files converted from arrow and compressed with snappy by js library called parquet-wasm. Opening with pyarrow or r-arrow works fine. Parquet2jl version `0.2.24` (also earlier versions do not work too). Julia 1.10.0.

```julia
ThriftInvalidFinalReadState: while attempting to read Parquet2.Metadata.Statistics, final index state was 7, expect 0.
Stacktrace:
  [1] readshort(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.Statistics}}, k::Int64, t::Int64)
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:22
  [2] read(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.Statistics}})
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:78
  [3] read
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:29 [inlined]
  [4] _readfield
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:140 [inlined]
  [5] readshortfield
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:158 [inlined]
  [6] readshort(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.ColumnMetaData}}, k::Int64, t::Int64)
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:15
  [7] read(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.ColumnMetaData}})
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:78
  [8] read
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:29 [inlined]
  [9] _readfield
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:140 [inlined]
 [10] readshortfield
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:158 [inlined]
 [11] readshort(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.Column}}, k::Int64, t::Int64)
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:15
 [12] read(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.Column}})
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:78
 [13] _read_setlike_elements(p::Thrift2.CompactProtocol{IOBuffer}, s::Int32, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.Column}})
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:44
 [14] read(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftList{Thrift2.ThriftStruct{Parquet2.Metadata.Column}}})
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:63
 [15] read
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:29 [inlined]
 [16] _readfield
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:140 [inlined]
 [17] readshortfield
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:158 [inlined]
 [18] readshort(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.RowGroup}}, k::Int64, t::Int64)
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:15
 [19] read(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.RowGroup}})
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:78
 [20] _read_setlike_elements(p::Thrift2.CompactProtocol{IOBuffer}, s::UInt8, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.RowGroup}})
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:44
 [21] read(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftList{Thrift2.ThriftStruct{Parquet2.Metadata.RowGroup}}})
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:63
 [22] read
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:29 [inlined]
 [23] _readfield
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:140 [inlined]
 [24] readshortfield
    @ Thrift2 C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\read.jl:158 [inlined]
 [25] readshort(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.FileMetaData}}, k::Int64, t::Int64)
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:15
 [26] read(p::Thrift2.CompactProtocol{IOBuffer}, ::Type{Thrift2.ThriftStruct{Parquet2.Metadata.FileMetaData}})
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:78
 [27] read
    @ Parquet2.Metadata C:\Users\pavlo\.julia\packages\Thrift2\HMxFD\src\codegen.jl:80 [inlined]
 [28] readmeta(v::Vector{UInt8}; check::Bool)
    @ Parquet2 C:\Users\pavlo\.julia\packages\Parquet2\JMLqR\src\dataset.jl:289
 [29] readmeta
    @ Parquet2 C:\Users\pavlo\.julia\packages\Parquet2\JMLqR\src\dataset.jl:283 [inlined]
 [30] Dataset(fm::Parquet2.FileManager{FilePathsBase.WindowsPath})
    @ Parquet2 C:\Users\pavlo\.julia\packages\Parquet2\JMLqR\src\dataset.jl:98
 [31] Dataset(p::FilePathsBase.WindowsPath; kw::@Kwargs{})
    @ Parquet2 C:\Users\pavlo\.julia\packages\Parquet2\JMLqR\src\dataset.jl:112
 [32] Dataset(p::FilePathsBase.WindowsPath)
    @ Parquet2 C:\Users\pavlo\.julia\packages\Parquet2\JMLqR\src\dataset.jl:110
 [33] Dataset(p::String; kw::@Kwargs{})
    @ Parquet2 C:\Users\pavlo\.julia\packages\Parquet2\JMLqR\src\dataset.jl:116
 [34] top-level scope

```

I can share the sample parquet file, but it’s not possible to include it.

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [July 5, 2024, 3:18pm UTC](https://discourse.julialang.org/t/parquet2jl-thriftinvalidfinalreadstate-while-attempting-to-read-parquet2-metadata-statistics/116668/2 "2024-07-05T15:18:05Z")

</div>

Tagging @ExpandingMan who is the maintainer

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [July 5, 2024, 4:19pm UTC](https://discourse.julialang.org/t/parquet2jl-thriftinvalidfinalreadstate-while-attempting-to-read-parquet2-metadata-statistics/116668/3 "2024-07-05T16:19:03Z")

</div>

If you can open an issue in the repo with the sample I can take a look.

---

<div class="post-metadata">

**Author:** ![PavloPolovyi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavlopolovyi/32/210479_2.png) [@PavloPolovyi](https://discourse.julialang.org/u/PavloPolovyi)\
**Post date:** [July 8, 2024, 9:37am UTC](https://discourse.julialang.org/t/parquet2jl-thriftinvalidfinalreadstate-while-attempting-to-read-parquet2-metadata-statistics/116668/4 "2024-07-08T09:37:36Z")

</div>

Thanks, I have opened issue in the repo

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [July 9, 2024, 10:02pm UTC](https://discourse.julialang.org/t/parquet2jl-thriftinvalidfinalreadstate-while-attempting-to-read-parquet2-metadata-statistics/116668/5 "2024-07-09T22:02:28Z")

</div>

Thanks. It appears there was a metadata update last November that I never included. I’m pretty sure my thrift implementation is not supposed to break when that happens, so it probably has a bug, but it wasn’t clear to me how to fix it and I haven’t tried that hard yet. I did of course update the parquet metadata schema though, so it’s fixed in parquet.

Anyway, you probably saw this relatively early because sounds like you are using a newer writer that probably got updated relatively early. Regardless, if anyone else was seeing similar issues, they should be resolved for you as well as soon as `0.2.25` is tagged.