[ANN] CSV.jl 1.0 Release

[quinnj]

Hello!

We’re pleased to present CSV.jl 1.0!

TLDR: this is a pretty substantial rewrite of the parsing and writing internals, with big performance improvements and a few new features. The main entry points remain the same: CSV.File, CSV.read, CSV.Rows, CSV.Chunks, CSV.write, and CSV.RowWriter. There are some changes to defaults and old keyword arguments, so there’s a migration guide to help with upgrading.

Highlights include:

• New parsing internals. The parser first uses SIMD & other specialized vector instructions to build an index of row and field boundaries, taking quotes and escapes into account. Typed parsing then works from that index in parallel. The readers share this machinery, so fixes to quote handling, row boundaries, and recovery apply across the package.
• A new CSV.lazy reader. Index a source once, then parse cells as you access them. lazyfile[2, :amount] parses one cell; lazyfile[:amount] returns a column of text. CSV.File(lazyfile) does the full typed parse without repeating the structural scan.
• New string storage. Inferred text uses DataStrings.DataString, an AbstractString that stores short strings inline and longer strings in column-owned buffers. The design is actually based on the arrow data format string storage. stringtype=String and explicit InlineStrings types remain available.
• Better handling of parsing problems. Instead of a warning for every bad value, you get one summary and structured records from CSV.problems(file): row, column, byte position, kind, and message. on_error=:collect records problems silently; on_error=:error throws CSV.ParseError.
• Tables.Scan support. Select, rename, convert, filter, offset, and limit inside the reader. Columns you don’t select aren’t sampled or parsed, and rows rejected by the filter don’t need their remaining columns parsed.
• Higher-precision date-times. Date-time columns normally infer as Timestamp{Nanosecond} from Durations.jl, preserving fractions that DateTime cannot represent. You can still request types=DateTime; sub-millisecond values then report a parsing problem instead of silently losing precision.
• A new parallel writer. Parallel writing, deterministic output across thread counts, explicit quoting options, Printf-style floatformat, and automatic gzip output for .gz paths. CSV.write and CSV.RowWriter share the same value-formatting code.
• Julia 1.10+ and trim compilation. The package now requires Julia 1.10 or later. Trim-compiled workloads are part of the test suite so this remains something we test and maintain going forward.
• Resolution of many open issues An apology to all those who have been waiting on open issues for a while. My time is spread across a lot of projects and I can’t always keep up with everything. Part of the goal of the internals rewrite & 1.0 release was to collectively fix a number of long-standing issues.

Performance

One of my main goals was improving performance without having a different set of parsing rules for every reader. Here are measurements on twelve synthetic CSV shapes, covering numeric data, strings, quoted fields, dates, wide tables, and missing values.

These are approximately 200 MiB files on an Apple M3 Max, using Julia 1.12.6, CSV.jl 0.10.17 and 1.0.0, Polars 1.44.2, PyArrow 25.0.1, and DuckDB 1.5.5. Each operation gets a warm-up, followed by four timed runs; the charts show the fastest. Each engine infers its own types, and reads produce fully materialized columns.

Compared with CSV.jl 0.10.17, the read speedups range from 1.6–9.8× across these cases. Write speedups range from 2.2–12.8× with one thread and 7.9–58.3× with eight. These are elapsed-time ratios. The old writer was effectively serial, so parallel writing makes a big difference here.

Compared with the other libraries, CSV.jl does particularly well on string-heavy data. At eight threads, plain string columns read at about 2,922 MiB/s versus 985 for Polars, and quoted strings at 2,131 versus 1,416. There is plenty of variation by workload, which is why I wanted to show the full set. Small gaps are best treated as ties.

There are a few important differences in what these calls do. DuckDB’s read timings include converting the query result to an Arrow table; they aren’t measurements of an in-database query. Date/time types also differ: PyArrow inferred seconds, Polars and DuckDB microseconds, and CSV.jl 1.0 nanoseconds. The quoted input contains embedded newlines, so PyArrow uses newlines_in_values=True, which can reduce multithreaded read performance. These results compare the tested calls and their inferred output types, rather than forcing identical representations everywhere.

Single-threaded reads

At eight threads, CSV.jl has the highest measured write throughput on the date/time and wide-table inputs in this run. There is still room to improve, and the fastest library depends on the input and thread count.

Single-threaded writes

Thread scaling: strings and mixed columns

CSV.jl 1.0 has its highest throughput at 14 threads for both inputs among the thread counts tested in this run. The charts show configured thread counts, not a guarantee that every operation uses every thread.

Measurement details and full tables

• Engines infer their own schemas; Polars uses try_parse_dates=True and infer_schema_length=10000. The low-cardinality input uses CSV.jl 1.0’s default, unpooled storage.
• Write timings start with materialized tables. Throughput uses each engine’s output file size; formatting and inferred types can produce different byte counts. Writes do not include an explicit disk flush (fsync).
• The 0.10-to-1.0 speedup chart uses elapsed-time ratios. The other bar charts and tables use MiB/s, where higher is better. Small differences can be timing noise.
• Input shapes come from the package’s benchmark matrix. These measurements use a separate comparison harness around each library’s file APIs.

Read, 1 thread, MiB/s (higher is better)

shape	CSV.jl 1.0	CSV.jl 0.10	polars	pyarrow	duckdb
numeric (int+float)	554	285	563	333	241
floats	588	313	628	578	292
mixed (int,float,str,date,bool)	607	231	544	364	228
strings (8 cols)	646	111	300	291	156
quoted strings *	446	151	387	267	169
long text	1585	482	1608	1744	539
wide (200 cols)	307	190	366	187	108
long+narrow (2 cols)	582	263	540	316	199
low-cardinality strings	522	155	322	274	126
date/datetime/time	686	70	52	444	145
bools	553	183	227	168	88
sparse (missing)	388	186	172	167	112

Read, 8 threads, MiB/s (higher is better)

shape	CSV.jl 1.0	CSV.jl 0.10	polars	pyarrow	duckdb
numeric (int+float)	3096	1328	3260	1750	247
floats	3295	1794	4272	3077	299
mixed (int,float,str,date,bool)	2690	1118	2998	2050	232
strings (8 cols)	2922	890	985	1813	162
quoted strings *	2131	1098	1416	941	190
long text	6626	2768	3033	6706	654
wide (200 cols)	1503	932	1400	973	144
long+narrow (2 cols)	2918	1275	4276	2176	249
low-cardinality strings	2778	622	2069	1744	173
date/datetime/time	3508	414	429	4065	290
bools	2861	919	3034	2021	180
sparse (missing)	2342	891	2142	1493	184

Write, 1 thread, MiB/s (higher is better)

shape	CSV.jl 1.0	CSV.jl 0.10	polars	pyarrow	duckdb
numeric (int+float)	442	106	602	247	171
mixed (int,float,str,date,bool)	420	53	515	387	213
strings (8 cols)	298	111	405	810	239
quoted strings	340	124	444	512	238
date/datetime/time	524	44	200	543	282
wide (200 cols)	306	83	325	115	96
long text	1820	810	2882	3594	638

Write, 8 threads, MiB/s (higher is better)

shape	CSV.jl 1.0	CSV.jl 0.10	polars	pyarrow	duckdb
numeric (int+float)	2648	128	3872	269	839
mixed (int,float,str,date,bool)	2279	80	2911	390	1096
strings (8 cols)	1839	116	2337	790	1166
quoted strings	1951	128	2556	515	1172
date/datetime/time	2626	49	1032	545	1524
wide (200 cols)	1742	60	1321	115	143
long text	5545	705	6551	3562	2308

Quoted fields include embedded newlines; see the PyArrow option note above. CSV.jl 1.0 values are bold for reference, not to mark the winner.

Breaking changes

This is a major version, so downstream packages need to opt in through compat bounds. A few changes are worth checking when you upgrade:

• Inferred text now uses DataStrings.DataString. Pass stringtype=String, or load InlineStrings and choose a type if a consumer needs a specific representation.
• Pooling is off by default. pool=(0.2, 500) restores the old policy.
• An unquoted empty field is always missing; a quoted empty field is empty text. The writer preserves that distinction.
• Boolean inference accepts true/True/TRUE and false/False/FALSE. Use truestrings and falsestrings for other spellings.
• Date-time inference preserves nanosecond precision where the values fit, as described above. The migration guide covers wider date ranges and explicit DateTime conversion.
• on_error=:collect replaces silencewarnings=true. Several old options are removed or renamed, and types, select, drop, and pool no longer accept functions.
• CSV.Rows, CSV.Chunks, and CSV.lazy retain source bytes and a full structural index. They avoid materializing all values at once, but they aren’t unbounded stream readers. Gzip and non-file sources are fully buffered, so check memory use for large inputs.

The migration guide has the full option mapping and memory behavior changes, and the release notes cover the rest. If something that used to work no longer does, please open an issue and we can figure out how to fix/support it. I’m very open to “unbreaking” anything that broke unintentionally.

Why 1.0 now?

I said in the 0.9 announcement that 1.0 was coming “in the very near future”. That took a little longer than planned

A shoutout to @drvi is due as we’ve bounced ideas on vectorized csv parsing for several years now and played with various prototypes/shapes/sizes/approaches. AI has been a huge help here in trying out various ideas and helping to rigorously test the correctness of things.

A note on the development process: this release involved substantial work with Codex and Claude, with me directing the design and reviewing the changes. The implementation went through repeated review and testing, including fuzzing and tests around quotes, malformed input, and chunk boundaries.

Thanks again to everyone who tested, filed issues, asked questions, or just expressed appreciation along the way. Please keep the feedback coming at the repo or the #data Slack channel. We look forward to hearing from you!

-Jacob Quinn & JuliaData maintainers

[mthelm85]

This is fantastic, Jacob! Thank you, and all of the contributors - I’ve been using CSV.jl for many years and I appreciate the amazing work you all have put into it!

[nilshg]

What a day… What’s next, Distributions 1.0???

Congrats on the release, and one thing that I think isn’t mentioned but which I just noticed in my current project is a nice speedup in TTFDataFrame - on 1.13 with 20 threads:

# 3 csvs, make 225,000-by-28 DataFrame
# CSV 0.10.17, two runs
  5.788769 seconds (14.33 M allocations: 939.174 MiB, 7.31% gc time, 31 lock conflicts, 468.94% compilation time: 35% of which was recompilation)
  0.162671 seconds (2.47 M allocations: 274.021 MiB, 20.62% gc time, 8 lock conflicts, 0.13% compilation time)

# CSV 1.0, two runs
  4.033014 seconds (5.77 M allocations: 633.485 MiB, 7.49% gc time, 341 lock conflicts, 94.73% compilation time: 25% of which was recompilation)
  0.222373 seconds (74.81 k allocations: 309.190 MiB, 50.95% gc time, 336 lock conflicts)

# 3 csvs, make 10,000,000-by-27 DataFrame
# CSV 0.10.17, two runs
  6.749250 seconds (104.92 M allocations: 8.682 GiB, 22.54% gc time, 31 lock conflicts, 57.11% compilation time: 2% of which was recompilation)
  6.614801 seconds (98.38 M allocations: 8.419 GiB, 32.23% gc time, 12 lock conflicts, 0.01% compilation time)

# CSV 1.0, two runs
  4.412537 seconds (12.01 M allocations: 9.808 GiB, 21.78% gc time, 1086 lock conflicts, 6.56% compilation time)
  5.709952 seconds (11.68 M allocations: 9.791 GiB, 30.51% gc time, 981 lock conflicts)