# \[ANN\] DLMReader: the most versatile Julia package for reading delimited files yet
**URL:** https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899
**Category:** Package Announcements
**Tags:** csv, inmemorydatasets
**Created:** [May 30, 2022, 6:02am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899 "2022-05-30T06:02:35Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [May 30, 2022, 6:02am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/1 "2022-05-30T06:02:35Z")
I am excited to announce `DLMReader`, a `Julia` package for reading delimited files.
# Introduction
[`DLMReader.jl`](https://github.com/sl-solution/DLMReader.jl) is a multithreaded package for reading delimited files, and it is designed for `Julia` 1.6+ (64bit OS). The package performance is a trade-off between:
1. Flexibility
2. Memory efficiency
3. Low compilation time
4. High performance
The package scans the entire file (when `limit` is set, it may scan the input file twice) to gather information about the delimited file structure. This helps `DLMReader` to scale well for huge files, however, this is less effective for small files. When information about the file structure is obtained, they are sent to an internal distributer. The distributer starts multiple threads (it uses one thread when `threads = false` or the file size is small) and distributes the input file among them. Each thread allocates `buffsize` bytes of memory and starts processing each line of its chunk. Each line is stored as a customised type, called “LineBuffer”, and is sent to the `line_informat` to be pre-processed for parsing. Each thread then searches for each field in the line that is being processing and sends the raw text of each field to its value `informat`er, and the result is passed to the main parser for parsing and storing. `DLMReader` uses the `Julia` base functions for parsing `Integer`s, `Real`s, `String`s, the `Dates` package for parsing `DateTime`s, and the `UUIDs` package for parsing `UUID`s.
# Features
`DLMReader.jl` has some interesting features which distinguish it from other packages for reading delimited files. In what follows, I will go through some of those features;
- **`Informats`** : The `DLMReader` package uses `informats` to call a class of functions on the raw text before parsing its value(s). This provides a flexible and extendable approach to parse values with special patterns. For instance, using the predefined informat `COMMA!` allows users to read a numeric column with “thousands separator” and/or the dollar sign, e.g. using this informat, the raw text like “`$12,000.00`” will be parsed as “`12000.00`”. Moreover, `informat`s support function composing, e.g. `COMMA! ∘ ACC!` parses “`$(12,000.00)`” as “`-12000.00`”, i.e. `ACC!` is first applied and then `COMMA!` is applied on its result.
- **Fixed-width text** : If users pass the columns locations via the `fixed` keyword argument, the package reads those columns as fixed-width format. For instance, passing `fixed = Dict(1=>1:1, 2=>2:2)` helps to parse “`10`” as “`[1,0]`”. Mixing fixed-width format and delimited format is also allowed.
- **Multiple observations per line** : The package allows reading more than one observation per line. This can be done by passing the `multiple_obs = true` keyword argument. The multithreading feature (plus some other features) will be switched off if this option is set.
- **Fast file writer** : The `DLMReader` package exploits the `byrow` function from [`InMemoryDatasets.jl`](https://github.com/sl-solution/InMemoryDatasets.jl) to write delimited files into disk. This enables `DLMReader` to convert values to string using multiple threads.
- **Alternative delimiters** : User can pass a vector of delimiters to the function. In this case, `filereader` treats any of the passed delimiters as field delimiter.
# Benchmarks
The following benchmarks present preliminary results for `DLMReader.jl`’s performance compared to the `polars` and `data.table` packages. The benchmarks are based on the [`db-benchmark`](https://github.com/h2oai/db-benchmark) repository, i.e. they are the time that each package spends reading each file in the aforementioned repository.
There are some remarks about the presented benchmarks, (see [`InMemoryDatasets` announcement](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197)):
- Each read is done once (this includes compilation time for `DLMReader.jl`)
- The reported times do not include converting columns to pooled vectors (the same for all solutions).
- I report the total time, and use `fail` when a solution cannot complete a task
- I use a Linux REHL7 machine with 16 cores and `128GB` of memory.
- The results are based on the latest version of the benchmarked packages + the latest PRs submitted to the `db-benchmark` project.
- The OS system cache is freed for `polars`. The reason for this is that `polars` exploits the OS file cache and this significantly improves the `polars` reading performance, however, this does not affect other solutions.
**The `groupby` task timing in seconds - smaller is better**
| Data | DLMReader | polars | DT |
| --- | --- | --- | --- |
| 1e7 - 2e0 | 18 | 15 | 11 |
| 1e7 - 1e1 | 19 | 13 | 5 |
| 1e7 - 1e2 | 19 | 15 | 3 |
| | | | |
| 1e8 - 2e0 | 29 | 168 | 132 |
| 1e8 - 1e1 | 31 | 167 | 83 |
| 1e8 - 1e2 | 32 | 169 | 53 |
| | | | |
| 1e9 - 2e0 | 173 | 1624 | 1766 |
| 1e9 - 1e1 | 174 | 1605 | 1137 |
| 1e9 - 1e2 | 153 | 1640 | 727 |
**The `join` task timing in seconds - smaller is better**
| Data | DLMReader | polars | DT |
| --- | --- | --- | --- |
| 1e7 | 26 | 28 | 32 |
| 1e8 | 61 | 304 | 441 |
| 1e9 | 269 | **fail** | **fail** |
---
### Author:  [@Storopoli](https://discourse.julialang.org/u/Storopoli)
#### Post date: [May 30, 2022, 4:18pm UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/2 "2022-05-30T16:18:23Z")
I don’t get it.
This is a package that reads delimited files, like `CSV.jl`.
But your benchmarks are for data manipulation stuff: join and groupby.
What am I getting wrong?
---
### Author:  [@vtomar](https://discourse.julialang.org/u/vtomar)
#### Post date: [May 31, 2022, 3:22am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/3 "2022-05-31T03:22:11Z")
Any reason for not benchmarking against CSV.jl?
---
### Author:  [@xinchin](https://discourse.julialang.org/u/xinchin)
#### Post date: [May 31, 2022, 11:22pm UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/4 "2022-05-31T23:22:39Z")
why user defined informats must be registered? why not directly use the function itself?
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [June 1, 2022, 12:25am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/5 "2022-06-01T00:25:59Z")
Same here,
I was expecting to view reading and writing times.
Maybe he means the time to read the databases used for the different tasks, i.e. with different sizes and missings, without any further processing.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 1, 2022, 8:02am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/6 "2022-06-01T08:02:41Z")
The benchmark is just about reading files - in db-benchmark repository there are multiple files which needed to be read and analysed, and the benchmarks in this thread only focus on reading csv files.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 1, 2022, 8:23am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/7 "2022-06-01T08:23:48Z")
> [@vtomar](#):
>
> Any reason for not benchmarking against CSV.jl?
I had included CSV.jl at some point, but I dropped it because:
- The CSV.jl is single threaded in the db-benchmark scripts, and it may not be fair to CSV.jl
- And, it is not a good idea to enable multi-threading in CSV.jl.
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [June 1, 2022, 8:27am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/8 "2022-06-01T08:27:23Z")
Why is it not a good idea to enable multi-threading in CSV.jl?
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 1, 2022, 8:28am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/9 "2022-06-01T08:28:21Z")
> [@xinchin](#):
>
> why user defined informats must be registered? why not directly use the function itself?
because DLMReader does not use the user defined function directly, it internally creates a new object from the passed function and sends that particular object to the informater. the `register_informat` function does this task.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 1, 2022, 8:37am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/10 "2022-06-01T08:37:27Z")
> [@Juan](#):
>
> Maybe he means the time to read the databases used for the different tasks, i.e. with different sizes and missings, without any further processing.
You are right;
- the proposed benchmarks are related to db-benchmark - there are some discussions to include a [reading](https://github.com/h2oai/db-benchmark/issues/131) task in those benchmark + since, I have used those benchmarks for `InMemoryDatasets`, I decided to reuse them for this announcement.
- There is no missing values in the benchmarked files - this would not change anything for `DLMReader`, since, the package allocates union of missing for every reading.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 1, 2022, 9:59am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/11 "2022-06-01T09:59:41Z")
> [@nilshg](#):
>
> Why is it not a good idea to enable multi-threading in CSV.jl?
- CSV.jl uses chained vector from SentinelArrays.jl, and random access in chained vectors is expensive, thus, it would slow down the subsequence operations on data sets.
- Enabling multi-threading for CSV.jl significantly increases the memory usage.
- Additionally, it slows down the importing files significantly for the benchmarks mentioned in this post.
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [June 1, 2022, 11:25pm UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/12 "2022-06-01T23:25:29Z")
DLMReader could be used together with InMemoryDatasets.jl to try to beat other solutions (Polars) on the full [Database-like ops benchmark](https://h2oai.github.io/db-benchmark/)
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 2, 2022, 7:32am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/13 "2022-06-02T07:32:35Z")
One interesting thing that I noticed during these benchmarkings is that for the `1e9 join` task, the combination of `InMemoryDatasets` and `DLMReader` finish reading and processing data long before other packages realise that they are failing in the reading part!
---
### Author:  [@xinchin](https://discourse.julialang.org/u/xinchin)
#### Post date: [June 2, 2022, 10:22pm UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/14 "2022-06-02T22:22:20Z")
I’m sure there are good reasons for registering informats, but I meant why `filereader` doesn’t do it automatically?
---
### Author:  [@xinchin](https://discourse.julialang.org/u/xinchin)
#### Post date: [June 2, 2022, 10:28pm UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/15 "2022-06-02T22:28:13Z")
I never been a fan of CSV.jl special array type for reading CSV Files but I never thought it degrades performance!!! 😮
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [June 4, 2022, 12:55am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/16 "2022-06-04T00:55:39Z")
it’s related with this:
> **[How to make your joins faster in DataFrames.jl?](https://bkamins.github.io/julialang/2021/07/30/joins.html)**
>
> Introduction
---
### Author:  [@xinchin](https://discourse.julialang.org/u/xinchin)
#### Post date: [June 6, 2022, 12:05am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/17 "2022-06-06T00:05:38Z")
> [@Juan](#):
>
> it’s related with this:
I’m confused?? 😕 I shouldn’t use inlinestring they’r slow?? how’s related to chain vector and random access?
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 7, 2022, 9:37am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/18 "2022-06-07T09:37:24Z")
> [@xinchin](#):
>
> I’m sure there are good reasons for registering informats, but I meant why `filereader` doesn’t do it automatically?
Registering an informat needs compilation, thus, if the `filereader` function does this, it triggers compilation every time (which is not ideal). However, this also means redefining a function would not change the definition of already registered infomrat.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 7, 2022, 9:49am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/19 "2022-06-07T09:49:27Z")
To be precise, my comment about chained vectors would not be an issue for small data sets, it would be a problem in scenarios where user works with large data sets (or in benchmarkings) and mostly for those operations which need random `getindex` (so using `InlineStrings` is ok - Actually, `DLMReader` supports `InlineStrings` out of the box )
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [June 9, 2022, 7:32am UTC](https://discourse.julialang.org/t/ann-dlmreader-the-most-versatile-julia-package-for-reading-delimited-files-yet/81899/20 "2022-06-09T07:32:26Z")
Due to [this issue](https://github.com/sl-solution/DLMReader.jl/issues/5) we are using `Parsers` for parsing Float64 and Float32.