# Package for reading/writing ~100GB data files

**URL:** <https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458>\
**Category:** General Usage\
**Created:** [November 13, 2018, 1:46am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458 "2018-11-13T01:46:59Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 13, 2018, 1:46am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/1 "2018-11-13T01:46:59Z")

</div>

Then, what is nowadays the fastest package/function/options to read and write say 100GB data with julia?  
For example with 10000 columns. Each column with all elements of the same type and a header.  
It would be great to be able to read and write directly compressed files.

I would like to find on that table the columns or the rows containing a certain value and read them to calculate something else, for example the median o perform a regression.  
Up to now I was using R’s data.table function fread for smaller sets and I had to use sqlite for larger datasets.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [November 13, 2018, 2:00am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/2 "2018-11-13T02:00:16Z")

</div>

I moved this to its own topic (instead of the 2-year old JuliaDB announcement thread)

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [November 13, 2018, 2:38am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/3 "2018-11-13T02:38:20Z")

</div>

That’s the last time i looked at it for CSV [Benchmarking ways to write/load DataFrames IndexedTables to disk](https://discourse.julialang.org/t/benchmarking-ways-to-write-load-dataframes-indexedtables-to-disk/8973)

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [November 13, 2018, 2:24am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/4 "2018-11-13T02:24:08Z")

</div>

R data.table will be able to deal with out-of-memory data in the near future so you do not need to use external database. In Julia, the current feasible way to read large data sets is JuliaDB.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [November 13, 2018, 10:23pm UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/5 "2018-11-13T22:23:02Z")

</div>

> [@Yifan\_Liu](#):
>
> R data.table will be able to deal with out-of-memory data in the near future

Where did you get this info? Also given there is no easy way to convert data from R to Julia. Does this help with reading data into Julia?

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [November 13, 2018, 10:51pm UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/6 "2018-11-13T22:51:21Z")

</div>

I chatted with a dev guy of RDatatable a couple of days ago, and he told me that the out-of-memory functionality will be available in the near future. Since data.table is written in C, I guess it can be transferred to Julia without too much hassle. Now they have a Python version of data.table

> **[GitHub - h2oai/datatable: A Python package for manipulating 2-dimensional...](https://github.com/h2oai/datatable)**
>
> A Python package for manipulating 2-dimensional tabular data structures - GitHub - h2oai/datatable: A Python package for manipulating 2-dimensional tabular data structures

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 14, 2018, 12:27am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/7 "2018-11-14T00:27:33Z")

</div>

Anyway I guess only some operations such as reading rows, columns, filtering will be available, like in a traditional database.  
The future of scientific computation will be to be able to perform all kind of computations directly on disk, for example a regression with random effects or MCMC with a dataset much larger than memory.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [November 14, 2018, 3:19am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/8 "2018-11-14T03:19:47Z")

</div>

> [@Juan](#):
>
> The future of scientific computation will be to be able to perform all kind of computations directly on disk

JuliaDB.jl is meant to have lots of online algorithms to do that.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 14, 2018, 6:36am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/9 "2018-11-14T06:36:12Z")

</div>

> [@Yifan\_Liu](#):
>
> Since data.table is written in C, I guess it can be transferred to Julia without too much hassle.

AFAIK `data.table` only needs to deal with the types allowed by R (integer, float, categorical, string, date, maybe I forgot something?). Writing an efficient implementation for a restricted number of bits types is considerably easier than handling generic types, which may or may not be bits types.

---

<div class="post-metadata">

**Author:** ![MA\_Laforge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ma_laforge/32/385_2.png) [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)\
**Post date:** [November 17, 2018, 1:39am UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/10 "2018-11-17T01:39:20Z")

</div>

My new prefered format for saving large datasets is HDF5. To read/write in this format in Julia, you can use the package found here:

> **[GitHub - JuliaIO/HDF5.jl: Save and load data in the HDF5 file format from...](https://github.com/JuliaIO/HDF5.jl)**
>
> Save and load data in the HDF5 file format from Julia

It can be installed with the following `add` statement:

`] pkg add HDF5`

It appears the HDF format was originally developed at the “National Center for Supercomputing Applications”, presumably for dealing with large amounts of data. You might find the Wiki page interesting:

> **[Hierarchical Data Format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format)**
>
> Hierarchical Data Format (HDF) is a set of file formats (HDF4, HDF5) designed to store and organize large amounts of data. Originally developed at the U.S. National Center for Supercomputing Applications, it is supported by The HDF Group, a non-profit corporation whose mission is to ensure continued development of HDF5 technologies and the continued accessibility of data stored in HDF.
> In keeping with this goal, the HDF libraries and associated tools are available under a liberal, BSD-like licen...

Note that HDF5 supports the use of compression algorithms, but I have not personally used them.

### Good to save matrices of a single data type

This format excels at writing large matrices of data (not typically what people refer to as “tables” of data, though). In other words, it is great if all elements of a “table” are of the same data type (ex: all `Float64`).

### Saving columns of different data types

I believe it is possible for HDF5 to natively define a table format with different types for different rows… but I personally have not used this feature much. That being said, I believe that what you call a “table” is called a “Compound Dataset” in HDF5-speak.

### Saving columns of different data types: A hack

Note that HDF5 is a Hierarchical data storage format… so, if you wish to save columns that each store data of different types, you could, in theory manually break down your “table” into rows that are each stored as different “subfolders” of your table. For example:

```julia-auto
mytable1\key: stores vector (int)
mytable1\name: stores vector (string)
mytable1\weight: stores vector (float64)
mytable1\height: stores vector (float64)
...

```

### Useful tools

The HDF group provides a viewer tool that lets you browse the contents of an HDF5 file. It basically makes the file format feel somewhat like a text file… because you can always look at the “raw” binary data with this tool:

[https://support.hdfgroup.org/products/java/hdfview/](https://support.hdfgroup.org/products/java/hdfview/)

You can even start seeding an HDF5 file using this GUI-based tool - or simply modify the contents of a pre-existing dataset.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 17, 2018, 1:10pm UTC](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458/11 "2018-11-17T13:10:02Z")

</div>

What if I want to fit a regression model with random effects (or a survival analysis) with data that doesn’t fit on memory?  
Can JuliaDB.jl do it, or maybe OnlineStats.jl or combining MixedModels with any of them?  
Some day I will open a new thread with an example to see different solutions.