# Suggestions for a package to read tabular data

**URL:** <https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071>\
**Category:** Data\
**Tags:** question\
**Created:** [February 13, 2017, 3:30am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071 "2017-02-13T03:30:55Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [February 13, 2017, 3:30am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/1 "2017-02-13T03:30:55Z")

</div>

I have thousands of \*.csv files to process, which contains table data. I usually use Python + Pandas. To get high performance, I tried to transfer from Python to Julia for some data processing work. I tested DataFrames.jl and Pandas, and found that the speed of Pandas is much higher. Is there any other package in Julia which can run a bit faster than Pandas?

 ![](https://global.discourse-cdn.com/julialang/original/3X/9/8/98c3a8612aecf0dc44cc75f11faaa3e56d71b71d.png)

I found there are several other packages to handle table data. Can anyone give me some suggestions on the proper package?  
DataFrames.jl  
TypedTables.jl  
DataTables.jl

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 13, 2017, 3:36am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/2 "2017-02-13T03:36:18Z")

</div>

CSV.jl?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 13, 2017, 3:37am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/3 "2017-02-13T03:37:07Z")

</div>

What about Pandas.jl?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 13, 2017, 3:37am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/4 "2017-02-13T03:37:59Z")

</div>

Out of curiosity, is this with the NullableArray backend?

---

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [February 13, 2017, 3:40am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/5 "2017-02-13T03:40:49Z")

</div>

Thanks! I have also tested CSV.jl, and it ~~is slower than~~ has similar speed as DataFrames.jl for my data (a little bit faster). I have thousands of files, and thus I care about the reading speed of loading the file.

---

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [February 13, 2017, 3:45am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/6 "2017-02-13T03:45:46Z")

</div>

Thanks for your suggestion. I will try this package.  
For DataFrames.jl, it runs on windows 7+ Julia 0.5 + DataFrames 0.8.5.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 13, 2017, 3:48am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/7 "2017-02-13T03:48:30Z")

</div>

Okay, you’re not using master. You may want to read this:

> [@Announcement: DataFrames Future Plans](https://discourse.julialang.org/t/announcement-dataframes-future-plans/266):
>
> UPDATE: the plan described below is going to be implemented in a different way from what was initially announced. The DataFrames package will remain the same as it is now: instead, the new framework will be provided by the new [DataTables](https://github.com/JuliaData/DataTables.jl) package. This will allow for a less disruptive migration of existing code relying on DataFrames. UPDATE 2: an updated summary is available [in this post](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646). Towards DataFrames 0.9.0 The DataFrames package and the surrounding ecosystem are currently undergoing a d…

This should be an update very soon which has a lot of changes, and one big change is increased performance. This may be worth holding out for.

---

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [February 13, 2017, 3:55am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/8 "2017-02-13T03:55:10Z")

</div>

Sounds exciting! I will wait for the new package.  
Once the CSV file is loaded, the speed row iteration of DataFrame.jl+Julia is much faster than Pandas+Python.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [February 13, 2017, 4:06am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/9 "2017-02-13T04:06:22Z")

</div>

Can you share your experience with CSV.jl? The package aims to be the fastest csv reader for Julia and in most cases, that has been the experience. I’d love to hear how it’s falling short for you. (Interested because I’m the primary package author 😃)

---

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [February 13, 2017, 4:44am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/10 "2017-02-13T04:44:33Z")

</div>

I tested the CSV.jl. It is a bit faster than DataFrames.jl for my data, but also much slower than Pandas+Python. It is very convenient to use.Thanks for your work.

 ![](https://global.discourse-cdn.com/julialang/original/3X/8/3/83a70bcc57428838224b7b7351f001e98a07a5ec.png)

---

<div class="post-metadata">

**Author:** ![andyferris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andyferris/32/235_2.png) [@andyferris](https://discourse.julialang.org/u/andyferris)\
**Post date:** [February 13, 2017, 5:14am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/11 "2017-02-13T05:14:00Z")

</div>

> Once the CSV file is loaded, the speed row iteration of DataFrame.jl+Julia is much faster than Pandas+Python.

In case you are interested, [TypedTables](https://github.com/FugroRoames/TypedTables.jl) is a package specifically designed for fast row iteration. Unfortunately, it requires the author to do a few small contortions to get it in the right format/type, extract the rows, use a functional workflow, etc, and is definitely less flexible that _DataFrames_, but I thought I might mention it in case speed is important and you are writing your own code to process rows.

(Note: _DataFrames_ used with care, or with one of the querying frameworks, can be similarly fast. When using either package, the thing that needs to be done right is to give the compiler the information of the concrete types stored in the table when it is processing the data. If you are iterating over rows yourself with _DataFrames_, you should get a large speedup from using a strongly-typed approach).

---

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [February 13, 2017, 5:22am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/12 "2017-02-13T05:22:04Z")

</div>

Thanks for your suggestion. I will take a look. The bottleneck for my code is the time of loading the data. There are thousands of files to process. The row-iteration operation of packages in Julia is very fast compared with Python+Pandas.

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [February 13, 2017, 8:55am UTC](https://discourse.julialang.org/t/suggestions-for-a-package-to-read-tabular-data/2071/13 "2017-02-13T08:55:12Z")

</div>

What is(/are) the source(s) for the CSV files?  
Do you know what format they are in? (MS Excel outputs RFC-4180 compliant files, MySQL by default does not.)  
It will be important to know what the character set encoding of the files is also (frequently Windows CP-1252 or ANSI Latin 1, or UTF-8) (unless you just have numerical data, in which case you won’t have to worry about different CSV variants or character set encoding).