Then, what is nowadays the fastest package/function/options to read and write say 100GB data with julia? For example with 10000 columns. Each column with all elements of the same type and a header. It would be great to be able to read and write directly compressed files. I would like to find on t…

I moved this to its own topic (instead of the 2-year old JuliaDB announcement thread)

R data.table will be able to deal with out-of-memory data in the near future so you do not need to use external database. In Julia, the current feasible way to read large data sets is JuliaDB.

[image] Yifan_Liu: R data.table will be able to deal with out-of-memory data in the near future Where did you get this info? Also given there is no easy way to convert data from R to Julia. Does this help with reading data into Julia?

I chatted with a dev guy of RDatatable a couple of days ago, and he told me that the out-of-memory functionality will be available in the near future. Since data.table is written in C, I guess it can be transferred to Julia without too much hassle. Now they have a Python version of data.table …

Anyway I guess only some operations such as reading rows, columns, filtering will be available, like in a traditional database. The future of scientific computation will be to be able to perform all kind of computations directly on disk, for example a regression with random effects or MCMC with a d…

[image] Juan: The future of scientific computation will be to be able to perform all kind of computations directly on disk JuliaDB.jl is meant to have lots of online algorithms to do that.

[image] Yifan_Liu: Since data.table is written in C, I guess it can be transferred to Julia without too much hassle. AFAIK data.table only needs to deal with the types allowed by R (integer, float, categorical, string, date, maybe I forgot something?). Writing an efficient implementation for …

My new prefered format for saving large datasets is HDF5. To read/write in this format in Julia, you can use the package found here: [image] GitHub - JuliaIO/HDF5.jl: Save and load data in the HDF5 file format from... Save and load data in the HDF5 file format from Julia …

What if I want to fit a regression model with random effects (or a survival analysis) with data that doesn’t fit on memory? Can JuliaDB.jl do it, or maybe OnlineStats.jl or combining MixedModels with any of them? Some day I will open a new thread with an example to see different solutions.

Package for reading/writing ~100GB data files

General Usage

xiaodai November 13, 2018, 2:38am 3

That’s the last time i looked at it for CSV Benchmarking ways to write/load DataFrames IndexedTables to disk

Topic		Replies	Views
JuliaDB out-of-memory computations New to Julia	2	537	December 6, 2018
Using JuliaDB to create larger than memory datasets and work with them? General Usage	3	1086	October 15, 2019
Importing big data General Usage question	21	5585	November 14, 2017
ANN: JuliaDB.jl Community	40	10053	November 13, 2018
Julia cookbook available New to Julia	13	1532	April 22, 2019

Package for reading/writing ~100GB data files

Related topics