Package for reading/writing ~100GB data files

R data.table will be able to deal with out-of-memory data in the near future so you do not need to use external database. In Julia, the current feasible way to read large data sets is JuliaDB.

1 Like