# CSV.jl "corrupts" data when a field is very large

**URL:** <https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513>\
**Category:** Data\
**Tags:** csv\
**Created:** [October 29, 2025, 1:31pm UTC](https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513 "2025-10-29T13:31:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [October 29, 2025, 1:31pm UTC](https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513/1 "2025-10-29T13:31:30Z")

</div>

I have a CSV (from [here](https://zenodo.org/records/8269680/files/MYRIAD-HES.zip?download=1)) where one field is a MULTIPOLYGON that can be very large, and when I import it with CSV (even a smaller 10 rows version) I got a strange corruption:

```julia-auto
julia> data = CSV.read("MYRIAD-HES/test.csv",DataFrames.DataFrame;delim=',',quotechar='\"')
9×8 DataFrame
 Row │ Event Hazard code starttime endtime Intensity Unit Geometry                          
     │ String7 String15 String15 Date Date Float64 String15 String                            
─────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ event0 heatwave hw31 2004-01-04 2004-01-06 294.0 Kelvin MULTIPOLYGON (((10.992 11.493, 1…
   2 │ event0 wildfire wf481347 2004-01-06 2004-01-17 8.46256 Area POLYGON ((11.161 10.799, 11.157 …
   3 │ event1 heatwave hw18 2004-01-04 2004-01-06 295.0 Kelvin POLYGON ((4.497 13.49, 4.497 12.…
   4 │ event1 wildfire wf479859 2004-01-05 2004-01-17 104.733 Area POLYGON ((4.388 13.44, 4.383 13.…
   5 │ event2 coldwave cw56 2004-01-04 2004-01-11 228.0 Kelvin MULTIPOLYGON (((-69.951 47.182, …
   6 │ event2 flood fl0 2004-01-04 2004-01-16 1.0 DFO severity LTIPOLYGON (((-91.514 34.569, -9…
   7 │ event3 flood fl0 2004-01-04 2004-01-16 1.0 DFO severity MULTIPOLYGON (((-91.514 34.569, …
   8 │ event3 extreme wind ew15 2004-01-07 2004-01-07 19.0 m/s POLYGON ((-83.442 44.935, -83.44…
   9 │ event4 flood fl0 2004-01-04 2004-01-16 1.0 DFO severity MULTIPOLYGON (((-91.514 34.569, …

julia> data[6,"Geometry"]
"LTIPOLYGON (((-91.514 34.569, -91.514 34.567, -91.516 34.567, -91.516 34.569, -91.514 34.569)), ((-91.518 34.569, -91.518 34.572, -91.516 34.572, -91.516 34.569, -91.518 34.569)), ((-91.505 34.576, -91.507 34.576, -91.507 34.578, -91.505 34.578, -91.505 34.576)), ((-91.502 34.581, -91.5 34.581, -91.5 34.578, -91.505 34.578, -91.505 34.583, -91.507 34.583, -91.507 34.585, -91.509 34.585, -91.509 34.587, -91.507 34.587, -91.507 34.59, -91.505 34.59, -91.505 34.587, -91.502 34.587, -91.502 34.581)), ((-91.493 34.635, -91.487 34.635, -91.487 34.632, -91.505 34.632, -91.505 34.635, -91.502 34.635, -91.502 34.637, -91.493 34.637, -91.493 34.635)), ((-91.498 34.655, -91.498 34.653, -91.5 34.653, -91.5 34.655, -91.498 34.655)), ((-91.502 34.659, -91.502 34.657, -91.505 34.657, -91.505 34.659, -91.502 34.659)), ((-91" ⋯ 429049 bytes ⋯ "8.488, -89.479 38.491, -89.474 38.491, -89.474 38.493, -89.477 38.493, -89.477 38.495, -89.472 38.495, -89.472 38.497, -89.474 38.497, -89.474 38.5, -89.468 38.5, -89.468 38.502, -89.461 38.502, -89.461 38.5, -89.454 38.5, -89.454 38.497, -89.45 38.497, -89.45 38.495, -89.461 38.495, -89.461 38.493, -89.463 38.493, -89.463 38.488, -89.477 38.488, -89.477 38.486, -89.47 38.486, -89.47 38.484, -89.477 38.484, -89.477 38.479, -89.479 38.479, -89.479 38.482)), ((-89.486 38.488, -89.486 38.491, -89.481 38.491, -89.481 38.488, -89.486 38.488)), ((-89.593 38.493, -89.596 38.493, -89.596 38.495, -89.593 38.495, -89.593 38.493)), ((-89.445 38.495, -89.447 38.495, -89.447 38.497, -89.445 38.497, -89.445 38.495)), ((-89.418 38.5, -89.418 38.497, -89.421 38.497, -89.421 38.5, -89.418 38.5)), ((-89.829 38.504, -89.834 3"

```

Note the “LTIPOLYGON” instead of “MULTIPOLYGON”  
I explorer the input csv file and it seems correct.

Is there other CSV import library? Or should I read the file manually ?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 29, 2025, 1:41pm UTC](https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513/2 "2025-10-29T13:41:40Z")

</div>

Please file an issue!

[https://github.com/JuliaData/CSV.jl/issues/new](https://github.com/JuliaData/CSV.jl/issues/new)

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [October 29, 2025, 1:51pm UTC](https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513/3 "2025-10-29T13:51:46Z")

</div>

done:

> <https://github.com/JuliaData/CSV.jl/issues/1171>
>
> I have a CSV file (43GB original version from \[here\](https://zenodo.org/records/…8269680/files/MYRIAD-HES.zip?download=1) and \[here\](https://nc.beta-lorraine.fr/s/nfxfykaHo9QRykT/download) a reduced 10MB, 10 rows version, produced with \`head -n10\`) where one field is a MULTIPOLYGON that can be very large, and when I import it with CSV.jl (even a smaller 10 rows version) I got a strange corruption:
> 
> \`\`\`
> julia\> data = CSV.read("MYRIAD-HES/test.csv",DataFrames.DataFrame;delim=',',quotechar='\\"')
> 9×8 DataFrame
> Row │ Event Hazard code starttime endtime Intensity Unit Geometry                          
> │ String7 String15 String15 Date Date Float64 String15 String                            
> ─────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
> 1 │ event0 heatwave hw31 2004-01-04 2004-01-06 294.0 Kelvin MULTIPOLYGON (((10.992 11.493, 1…
> 2 │ event0 wildfire wf481347 2004-01-06 2004-01-17 8.46256 Area POLYGON ((11.161 10.799, 11.157 …
> 3 │ event1 heatwave hw18 2004-01-04 2004-01-06 295.0 Kelvin POLYGON ((4.497 13.49, 4.497 12.…
> 4 │ event1 wildfire wf479859 2004-01-05 2004-01-17 104.733 Area POLYGON ((4.388 13.44, 4.383 13.…
> 5 │ event2 coldwave cw56 2004-01-04 2004-01-11 228.0 Kelvin MULTIPOLYGON (((-69.951 47.182, …
> 6 │ event2 flood fl0 2004-01-04 2004-01-16 1.0 DFO severity LTIPOLYGON (((-91.514 34.569, -9…
> 7 │ event3 flood fl0 2004-01-04 2004-01-16 1.0 DFO severity MULTIPOLYGON (((-91.514 34.569, …
> 8 │ event3 extreme wind ew15 2004-01-07 2004-01-07 19.0 m/s POLYGON ((-83.442 44.935, -83.44…
> 9 │ event4 flood fl0 2004-01-04 2004-01-16 1.0 DFO severity MULTIPOLYGON (((-91.514 34.569, …
> 
> julia\> data\[6,"Geometry"\]
> "LTIPOLYGON (((-91.514 34.569, -91.514 34.567, -91.516 34.567, -91.516 34.569, -91.514 34.569)), ((-91.518 34.569, -91.518 34.572, -91.516 34.572, -91.516 34.569, -91.518 34.569)), ((-91.505 34.576, -91.507 34.576, -91.507 34.578, -91.505 34.578, -91.505 34.576)), ((-91.502 34.581, -91.5 34.581, -91.5 34.578, -91.505 34.578, -91.505 34.583, -91.507 34.583, -91.507 34.585, -91.509 34.585, -91.509 34.587, -91.507 34.587, -91.507 34.59, -91.505 34.59, -91.505 34.587, -91.502 34.587, -91.502 34.581)), ((-91.493 34.635, -91.487 34.635, -91.487 34.632, -91.505 34.632, -91.505 34.635, -91.502 34.635, -91.502 34.637, -91.493 34.637, -91.493 34.635)), ((-91.498 34.655, -91.498 34.653, -91.5 34.653, -91.5 34.655, -91.498 34.655)), ((-91.502 34.659, -91.502 34.657, -91.505 34.657, -91.505 34.659, -91.502 34.659)), ((-91" ⋯ 429049 bytes ⋯ "8.488, -89.479 38.491, -89.474 38.491, -89.474 38.493, -89.477 38.493, -89.477 38.495, -89.472 38.495, -89.472 38.497, -89.474 38.497, -89.474 38.5, -89.468 38.5, -89.468 38.502, -89.461 38.502, -89.461 38.5, -89.454 38.5, -89.454 38.497, -89.45 38.497, -89.45 38.495, -89.461 38.495, -89.461 38.493, -89.463 38.493, -89.463 38.488, -89.477 38.488, -89.477 38.486, -89.47 38.486, -89.47 38.484, -89.477 38.484, -89.477 38.479, -89.479 38.479, -89.479 38.482)), ((-89.486 38.488, -89.486 38.491, -89.481 38.491, -89.481 38.488, -89.486 38.488)), ((-89.593 38.493, -89.596 38.493, -89.596 38.495, -89.593 38.495, -89.593 38.493)), ((-89.445 38.495, -89.447 38.495, -89.447 38.497, -89.445 38.497, -89.445 38.495)), ((-89.418 38.5, -89.418 38.497, -89.421 38.497, -89.421 38.5, -89.418 38.5)), ((-89.829 38.504, -89.834 3"
> \`\`\`
> 
> Note the "LTIPOLYGON" instead of "MULTIPOLYGON"
> I explorer the input csv file and it seems correct. Also, it works with CSVFiles.jl

(ps: it seems to work with CSVFiles.jl)

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [October 29, 2025, 1:53pm UTC](https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513/4 "2025-10-29T13:53:34Z")

</div>

> [@sylvaticus](#):
>
> Is there other CSV import library?

Yes. For example [GitHub - sl-solution/DLMReader.jl: High-performance delimited-file reader and writer for Julia](https://github.com/sl-solution/DLMReader.jl) , often used together with [GitHub - sl-solution/InMemoryDatasets.jl: Multithreaded package for working with tabular data in Julia](https://github.com/sl-solution/InMemoryDatasets.jl) .

---

<div class="post-metadata">

**Author:** ![TimG](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@TimG](https://discourse.julialang.org/u/TimG)\
**Post date:** [October 29, 2025, 2:11pm UTC](https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513/5 "2025-10-29T14:11:13Z")

</div>

Here’s another possible option:

> [@CSV.jl writing quoted strings](https://discourse.julialang.org/t/csv-jl-writing-quoted-strings/123989/15):
>
> For a CSV reader/writer robust in edgecases like that, you may try duckdb – convenient to use from Julia though [QuackIO.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/QuackIO).

Also, have you tried using CSV with `ntasks=1`. Multi-threading in CSV sometimes has some issues (e.g. with a `\n` character in a quoted string) and working single threaded can avoid these.

---

<div class="post-metadata">

**Author:** ![technocrat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/technocrat/32/220947_2.png) [@technocrat](https://discourse.julialang.org/u/technocrat)\
**Post date:** [October 29, 2025, 9:22pm UTC](https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513/6 "2025-10-29T21:22:52Z")

</div>

Try GeoDataFrames package.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 30, 2025, 1:06pm UTC](https://discourse.julialang.org/t/csv-jl-corrupts-data-when-a-field-is-very-large/133513/7 "2025-10-30T13:06:08Z")

</div>

Yeah, the fundamental problem here is that you are trying to read what is essentially a shapefile as a .csv. Maybe CSV.jl should handle this case but its definitely never going to be the right tool for the job.