# Julia import large out-of-memory csv data

**URL:** <https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532>\
**Category:** General Usage\
**Tags:** question\
**Created:** [June 29, 2017, 4:42am UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532 "2017-06-29T04:42:27Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [June 29, 2017, 4:42am UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/1 "2017-06-29T04:42:27Z")

</div>

I am a new Julia user and a PhD in Finance, and I would like to use Julia in my future financial research. I have been using R for most of my research projects.

I was wondering if Julia has a package similar to Sparklyr in R that could handle large out of memory data. My data is 20 GB in csv format, and my ram is 16GB.

I installed HPAT package in Julia, but I am not sure if it helps to handle big data. In addition, I noticed that there is a Spark package in Julia, does it have any function that enables me to import local data, like the spark\_read\_csv function in Sparklyr?

By the way, I want to know if I can turn on the bracket and quote autocomplete option in JuliaPro.

Also, I am not sure if I can use LaTex in Julia, since it is an important tool for academic publication.

Finally, I noticed that Wharton Research Data Services (the main data vendor for financial scholars) does not have any info in terms of accessing data via Julia, I have requested them to make a tutorial of this, hopefully this will be done before the end of July.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [June 29, 2017, 4:55am UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/2 "2017-06-29T04:55:43Z")

</div>

You might want to check out JuliaDB: [https://github.com/JuliaComputing/JuliaDB.jl](https://github.com/JuliaComputing/JuliaDB.jl) It’s very new, but it’s being actively worked on by some of the core Julia devs, so it has a pretty promising future.

Also, can you explain what you mean by using LaTeX in Julia? You can type various mathematical symbols using their LaTeX abbreviations with `\alpha<tab>` to get `α` and `\in<tab>` for `∈` and so on. This works in the Julia REPL, inside JuliaPro, in Juno, in Sublime with the Julia language package, and probably many other environments. Is that what you mean?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 29, 2017, 6:20am UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/3 "2017-06-29T06:20:17Z")

</div>

I second @rdeits’s suggestion for JuliaDB. If that does not work though, you can try reading and parsing line by line, storing in a compact format in memory. This is what I am currently doing for a 500GB dataset.

Regarding LaTeX, if you mean exporting result tables (eg regressions) like in R, I am not aware of any package. It could be a nice beginner project.

---

<div class="post-metadata">

**Author:** ![Daneel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daneel/32/175_2.png) [@Daneel](https://discourse.julialang.org/u/Daneel)\
**Post date:** [June 29, 2017, 7:10am UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/4 "2017-06-29T07:10:49Z")

</div>

If it’s plotting with LaTeX annotation you’re asking about then that is also possible.

---

<div class="post-metadata">

**Author:** ![ValdarT](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/valdart/32/24146_2.png) [@ValdarT](https://discourse.julialang.org/u/ValdarT)\
**Post date:** [June 29, 2017, 8:30am UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/5 "2017-06-29T08:30:41Z")

</div>

> [@Yifan\_Liu](#):
>
> Also, I am not sure if I can use LaTex in Julia, since it is an important tool for academic publication.

I’m also not entirely sure what you meant by this but perhaps [Weave.jl](http://weavejl.mpastell.com/stable/) is what you’re looking for.

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [June 29, 2017, 8:54am UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/6 "2017-06-29T08:54:24Z")

</div>

You can use LaTeX in your plots via [LaTeXStrings]([https://github.com/stevengj/LaTeXStrings.jl\_[LaTeXStrings](https://github.com/stevengj/LaTeXStrings.jl)](https://github.com/stevengj/LaTeXStrings.jl_%5BLaTeXStrings%5D(https://github.com/stevengj/LaTeXStrings.jl)). See [here](https://lectures.quantecon.org/jl/julia_plots.html) for an example (about a quarter of the way down the page).

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [June 29, 2017, 10:36am UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/7 "2017-06-29T10:36:42Z")

</div>

I mean that I need to transform tables and regression results in Julia to LaTex code and copy the code to Tex files, like what xtable and stargazer in R.

> **[xtable: Export Tables to LaTeX or HTML](https://cran.r-project.org/web/packages/xtable/index.html)**
>
> Coerce data to LaTeX and HTML tables.

> **[Stargazer](https://www.jakeruss.com/cheatsheets/stargazer/)**
>
> economist and statistical programmer

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [June 29, 2017, 1:36pm UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/8 "2017-06-29T13:36:35Z")

</div>

Thanks. This is exactly what I am looking for.

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [June 29, 2017, 1:50pm UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/9 "2017-06-29T13:50:16Z")

</div>

> [@Tamas\_Papp](#):
>
> Regarding LaTeX, if you mean exporting result tables (eg regressions) like in R, I am not aware of any package. It could be a nice beginner project.

Just for curiosity, then what does Spark and HPAT packages do? They seem to be designed for handling big data, then do they have any function that could import large out of memory data？

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [June 29, 2017, 2:00pm UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/10 "2017-06-29T14:00:41Z")

</div>

One more question, do you know how to turn on the bracket and quote autocomplete option in Julia Juno Atom?

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [June 29, 2017, 2:20pm UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/11 "2017-06-29T14:20:57Z")

</div>

I am using the most recent version of Julia Pro, but could not install the JuliaDB package, I got the error message as below

> JuliaDB can’t be installed because it has no versions that support 0.5.2 of julia. You may need to update METADATA by running `Pkg.update()`

how do I resolve this issue?

---

<div class="post-metadata">

**Author:** ![Daneel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daneel/32/175_2.png) [@Daneel](https://discourse.julialang.org/u/Daneel)\
**Post date:** [June 29, 2017, 2:43pm UTC](https://discourse.julialang.org/t/julia-import-large-out-of-memory-csv-data/4532/12 "2017-06-29T14:43:57Z")

</div>

The REQUIRE file lists Julia 0.6- so I think you’d have to update to Julia Pro 0.6 if that’s possible.

[https://github.com/JuliaComputing/JuliaDB.jl/blob/master/REQUIRE](https://github.com/JuliaComputing/JuliaDB.jl/blob/master/REQUIRE)

The documentation also states that “[JuliaDB works on Julia 0.6 or higher.](http://juliadb.org/latest/#Getting-started-1)”