# Storing big data file for fast access?

**URL:** <https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992>\
**Category:** General Usage\
**Tags:** question, big-data, io\
**Created:** [February 10, 2021, 1:42pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992 "2021-02-10T13:42:22Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mihrits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihrits/32/34494_2.png) [@mihrits](https://discourse.julialang.org/u/mihrits)\
**Post date:** [February 10, 2021, 1:42pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/1 "2021-02-10T13:42:22Z")

</div>

I have a somewhat big .txt data file (~70 MB, ~1M lines, ~30 columns) that I plan to study during the next few months. Data are mostly floats, but also strings and integers. And I’ll probably work with more files like that in the future. That means I most likely have to read it in more than a couple of times.

What is the recommended way of handling repeated reading of big files this? I would guess it is reasonable to save it as something else than .txt, which would be easier for the computer to read. But I don’t know if the implementation of reading .csv files is fast enough or should it be more binary like.

I have very briefly looked at [JuliaDB](https://juliadb.juliadata.org/stable/), [HDF5](https://juliaio.github.io/HDF5.jl/stable/), and [FortranFiles](https://traktofon.github.io/FortranFiles.jl/stable/). At the moment, JuliaDB seems the most approriate.

I am open to recommendations and tips. 🙂

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 10, 2021, 2:05pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/2 "2021-02-10T14:05:38Z")

</div>

> [@mihrits](#):
>
> I have a somewhat big .txt data file (~70 MB, ~1M lines, ~30 columns) that I plan to study during the next few months. […] That means I most likely have to read it in more than a couple of times.

That’s not very big—it easily fits in memory. Are you launching Julia over and over? Why not just leave a Jupyter/IJulia notebook open for your analyses with the data loaded in memory all of the time? If you only relaunch it once every few days or every few weeks, the load time ceases to matter.

(You can use Revise.jl to re-load modules you are editing without restarting Julia.)

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [February 10, 2021, 2:05pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/3 "2021-02-10T14:05:58Z")

</div>

I’m using Arrow.jl for larger files (20Gb) and it works very well. I especially like that there is compression built into the package. I compress with lz4 which is reasonably fast.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 10, 2021, 2:24pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/4 "2021-02-10T14:24:02Z")

</div>

FWIW:

```julia
julia> using CSV

julia> df = DataFrame(rand(1_000_000, 30), :auto);

julia> CSV.write("out.csv", df);

julia> using BenchmarkTools

julia> @btime CSV.read("out.csv", DataFrame);
  1.824 s (930 allocations: 254.16 MiB)

```

The csv file created above is ~550MB large, so I would expect your file to be parsed more quickly. Also, this is just using one thread, so might benefit from starting Julia with more threads if available. I’d think your file should be parsed in less than a second using CSV.jl on any reasonably modern machine.

Also CSV.jl should handle txt files just fine (if they are formatted correctly), so no need to save as a csv separately.

That said, for really large files I second the Arrow recommendation.

---

<div class="post-metadata">

**Author:** ![StatisticalMouse](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statisticalmouse/32/43370_2.png) [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)\
**Post date:** [February 10, 2021, 3:12pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/5 "2021-02-10T15:12:27Z")

</div>

Arrow format doesn’t do compression, right?

## Relation to other projects

#### **What is the difference between Apache Arrow and Apache Parquet?**

Parquet is not a “runtime in-memory format”; in general, file formats almost always have to be deserialized into some in-memory data structure for processing. We intend for Arrow to be that in-memory data structure.

Parquet is a storage format designed for maximum space efficiency, using advanced compression and encoding techniques. It is ideal when wanting to minimize disk usage while storing gigabytes of data, or perhaps more. This efficiency comes at the cost of relatively expensive reading into memory, as Parquet data cannot be directly operated on but must be decoded in large chunks.

Conversely, Arrow is an in-memory format meant for direct and efficient use for computational purposes. Arrow data is not compressed (or only lightly so, when using dictionary encoding) but laid out in natural format for the CPU, so that data can be accessed at arbitrary places at full speed.

Therefore, Arrow and Parquet complement each other and are commonly used together in applications. Storing your data on disk using Parquet and reading it into memory in the Arrow f

---

<div class="post-metadata">

**Author:** ![StatisticalMouse](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statisticalmouse/32/43370_2.png) [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)\
**Post date:** [February 10, 2021, 3:13pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/6 "2021-02-10T15:13:49Z")

</div>

in the Arrow format will allow you to make the most of your computing hardware.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [February 10, 2021, 3:43pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/7 "2021-02-10T15:43:08Z")

</div>

As far as I understand the compression in the Arrow.jl package happens separate from the Arrow format using CodecLz4.jl ([see here](https://arrow.juliadata.org/dev/manual/#Arrow.write) and [here](https://github.com/JuliaIO/CodecLz4.jl)), but it is convenient since its just an argument in the `Arrow.write` function.

---

<div class="post-metadata">

**Author:** ![StatisticalMouse](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statisticalmouse/32/43370_2.png) [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)\
**Post date:** [February 10, 2021, 7:25pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/8 "2021-02-10T19:25:24Z")

</div>

You are certainly right, I learned something.  
A couple of months ago I was juggling a dataset on my Mac which had trouble fitting in memory. Perhaps this would have helped. I thought I’d want compression, but I thought it doesn’t exist. I was using Dataframes.jl.

---

<div class="post-metadata">

**Author:** ![StatisticalMouse](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statisticalmouse/32/43370_2.png) [@StatisticalMouse](https://discourse.julialang.org/u/StatisticalMouse)\
**Post date:** [February 10, 2021, 7:31pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992/9 "2021-02-10T19:31:15Z")

</div>

To clarify, I was thinking of using compressed column data in-memory to enable a larger amount of it to be held in memory. Is this only for disk files?