Reading compressed csv gzopen cuts file short

General Usage
, ,
1

I seem to have stumbled into some kind of bug. I am working with large compressed CSV files. I’m loading the data with:
mydata = gzopen(x->CSV.read(x, DataFrame), mydatafile)

Then I just get this warning:

┌ Warning: thread = 1 warning: only found 82 / 88 columns around data row: 545401. Filling remaining columns with `missing`
└ @ CSV ~/.julia/packages/CSV/LiiJM/src/file.jl:592
545401×88 DataFrame

except the file has 719913 rows. I tried running the command again from the command line, then it first gave me 534731 rows, then afterwards it would give me 0 rows. It seems to me somewhere in the pipeline where CSV is trying to read from the gzip stream it’s failing when it should only wait some more, something like that.

Julia 1.12.5

  [336ed68f] CSV v0.10.16
  [a93c6f00] DataFrames v1.8.2
  [92fee26a] GZip v0.7.1
2

Are you able to share the data? Does CSV read everything correctly when you unzip the data first?

3

If I do gzopen(read, ...) and pass that to CSV.read, then it’s fine.