# Package for tabular data

**URL:** <https://discourse.julialang.org/t/package-for-tabular-data/17849>\
**Category:** Data\
**Created:** [November 21, 2018, 9:11pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849 "2018-11-21T21:11:53Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 21, 2018, 9:11pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/1 "2018-11-21T21:11:53Z")

</div>

Then for somebody starting to use Julia… which of that many options should we use for our new projects if we want to work with tabular data with real numbers, categorical variables and missing data as fast as possible?

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [November 21, 2018, 9:57pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/2 "2018-11-21T21:57:20Z")

</div>

I think all you really need to deal with, in your case, would be DataFrames.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 22, 2018, 11:46am UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/3 "2018-11-22T11:46:16Z")

</div>

But many other are supposed to be faster. I need that and compatibility with the most common packages.

On R I was using data.table and matrices.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [November 22, 2018, 11:59am UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/4 "2018-11-22T11:59:21Z")

</div>

Have you tried DataFrames? What is your specific use case? What did you find lacking? If you just need to use tabular data, DataFrames is perfect.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [November 22, 2018, 12:31pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/5 "2018-11-22T12:31:27Z")

</div>

@Juan I need… compatibility with the most common packages.  
Do you mean other packages within the Julia ecosystem? Well Julia is highly composable. Which means that when you use DataFrames when you have read that data in it should be compatible with all the other packages.

Look at the short section on Composable at [https://www.julialang.org](https://www.julialang.org)

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 22, 2018, 4:31pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/6 "2018-11-22T16:31:49Z")

</div>

Yes I mean that, DataFrames is compatible with other packages within the Julia ecosystem, but it’s slow. Then I need something faster but also compatible with packages such as GLM, Plot, MixedModels… I mean that I could feed their inputs directly with data with the chosen structure/framework.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 22, 2018, 5:22pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/7 "2018-11-22T17:22:45Z")

</div>

Common interfaces such as

> **[GitHub - JuliaData/Tables.jl: An interface for tables in Julia](https://github.com/JuliaData/Tables.jl)**
>
> An interface for tables in Julia. Contribute to JuliaData/Tables.jl development by creating an account on GitHub.

may eventually do this seamlessly, but currently that is work in progress.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 22, 2018, 6:14pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/8 "2018-11-22T18:14:56Z")

</div>

Will it be faster than dataframes? Will its syntax be much more complex/verbose?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 22, 2018, 6:20pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/9 "2018-11-22T18:20:06Z")

</div>

you haven’t explained exactly what operations are slow on DataFrames, or shown us the code you are using to perform the operations.

We might be able to help you improve performance if you give us something more concrete we can help you with.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [November 23, 2018, 4:34am UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/10 "2018-11-23T04:34:57Z")

</div>

Do you mean `using DataFrames`, ie loading the package? Yes, this is indeed slow at the moment. The solution will be a near-future version of the julia compiler that will be able to pre-compile, store, and load packages.

After the one-time load cost, DataFrames seem pretty fast and efficient.

regards,

/iaw

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 23, 2018, 1:36pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/11 "2018-11-23T13:36:09Z")

</div>

No, I mean using DataFrames is slow, traversing its elements, and working with them.  
I will create a MWE and post it here to try several things.

For now we can see other’s benchmarks. For example StaticArrays can be much faster than dataframes:

> **[GitHub - JuliaArrays/StaticArrays.jl: Statically sized arrays for Julia](https://github.com/JuliaArrays/StaticArrays.jl)**
>
> Statically sized arrays for Julia. Contribute to JuliaArrays/StaticArrays.jl development by creating an account on GitHub.

```julia
============================================
    Benchmarks for 3×3 Float64 matrices
============================================

Matrix multiplication -> 8.2x speedup
Matrix multiplication (mutating) -> 3.1x speedup
Matrix addition -> 45x speedup
Matrix addition (mutating) -> 5.1x speedup
Matrix determinant -> 170x speedup
Matrix inverse -> 125x speedup
Matrix symmetric eigendecomposition -> 82x speedup
Matrix Cholesky decomposition -> 23.6x speedup

```

That means that dataframes can be very slow.  
It would be great to see other benchmarks comparing speeds for different sizes and type of elements and some recommendations about what to use for each situation.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [November 23, 2018, 1:47pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/12 "2018-11-23T13:47:08Z")

</div>

`StaticArray`s and `DataFrame`s are completely different things, one can’t really compare them.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 23, 2018, 1:50pm UTC](https://discourse.julialang.org/t/package-for-tabular-data/17849/13 "2018-11-23T13:50:35Z")

</div>

> [@Juan](#):
>
> using DataFrames is slow

If you want to get help on this, it is best to post an MWE.