# Future directions for DataFrames.jl

**URL:** <https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247>\
**Category:** Data\
**Tags:** package, dataframes\
**Created:** [August 12, 2021, 6:23am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247 "2021-08-12T06:23:11Z")\
**Posts on this page:** 8\
**Page:** 3

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [August 19, 2021, 7:15am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/41 "2021-08-19T07:15:22Z")

</div>

> [@pdeffebach](#):
>
> [adding metadata to data frames](https://github.com/JuliaData/DataFrames.jl/pull/1458)

What do you think about the [solution](https://github.com/gcalderone/ReusePatterns.jl/blob/master/README.md#adding-metadata-to-a-dataframe-object) proposed by @gcalderone? Linking also his [post](https://discourse.julialang.org/t/how-to-add-metadata-info-to-a-dataframe/11168/95).

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [August 19, 2021, 10:24am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/42 "2021-08-19T10:24:23Z")

</div>

> [@rafael.guerra](#):
>
> What do you think about the [solution](https://github.com/gcalderone/ReusePatterns.jl/blob/master/README.md#adding-metadata-to-a-dataframe-object) proposed by @gcalderone? Linking also his [post](https://discourse.julialang.org/t/how-to-add-metadata-info-to-a-dataframe/11168/95).

AFAICT the problem with that solution is that the meta-data will be lost after calling any function that is fowarded to the `DataFrame` method. The main point of having per-column metadata is that it should be preserved, e.g. when adding/removing/transforming columns, subsetting rows, joining, etc. Whether it’s done within the `DataFrame` implementation itself, or using another type, it requires special code in many places to do that.

([Metadata.jl](https://github.com/Tokazama/Metadata.jl) can already be used to attach metadata to a `DataFrame` object, without even changing its type s but it has the same problem.)

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 19, 2021, 3:58pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/43 "2021-08-19T15:58:16Z")

</div>

IMHO:  
The **“handling data larger than fits into RAM”** is a good candidate  
as a [Julia “ **Diversity and Inclusion initiates.** ”](https://julialang.org/diversity/current/)

If we try _“recruiting folks from underrepresented communities to learn Julia”_  
and they have only ~ 1G / 2G / 4G RAM  
Then the `DataFrame` is not the optimal knowledge for them.

in this case - the [SQLite.jl](https://github.com/JuliaDatabases/SQLite.jl) is better for teaching ( and “handling data larger than fits into RAM” )

---

<div class="post-metadata">

**Author:** ![davibarreira](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davibarreira/32/20270_2.png) [@davibarreira](https://discourse.julialang.org/u/davibarreira)\
**Post date:** [June 2, 2022, 4:09pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/44 "2022-06-02T16:09:46Z")

</div>

@bkamins , has there been any advances in this front for DataFrames.jl? I mean, regarding data larger than fits into RAM. I’m in need of it, so perhaps I can contribute somehow.

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [June 2, 2022, 8:09pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/45 "2022-06-02T20:09:01Z")

</div>

@bkamins, @krynju and I have been investigating how to achieve out-of-core compute capabilities with DataFrames by using Dagger’s `DTable`. The idea is that you would wrap your `DataFrame`s with `DTable`s, and then do operations on those `DTable`s, which would then automatically move data out to disk as necessary. This would allow doing operations on `DataFrame`s which would normally take more RAM than is available on your system.

This sounds slightly complicated, but it’s actually pretty simple to do in practice, and I’ll be publishing a blog post in the next few weeks showing off how this all works and how to use it. The feature work for the out-of-core portion is basically done, just needing tests and docs before I merge it.

@krynju is actively working on implementing the DataFrames mini-language (`select`, `transform`, etc.) for the `DTable`, which will allow code that’s written for `DataFrame`s to work as-is with the `DTable` (and will also be imbued with the power of out-of-core).

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 2, 2022, 8:30pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/46 "2022-06-02T20:30:50Z")

</div>

Yes, I am very excited with @jpsamaroo + @krynju work and I hope by JuliaCon2022 there will be something available for user testing.

---

<div class="post-metadata">

**Author:** ![davibarreira](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davibarreira/32/20270_2.png) [@davibarreira](https://discourse.julialang.org/u/davibarreira)\
**Post date:** [June 2, 2022, 8:48pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/47 "2022-06-02T20:48:26Z")

</div>

Oh boy, that would be awesome! After using DataFrames.jl, I just find extremely painful to go back to python’s alternatives. I find DataFrames.jl just so intuitive. I can code without having to check for references on how to do something every minute or so.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [June 3, 2022, 4:23am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/48 "2022-06-03T04:23:27Z")

</div>

> [@Thibaut](#):
>
> Handling data larger than fits into RAM

i am working on this. An extension of the disk.frame brand from R. But it’s quite slow moving. Want to get the fundamental building blocks right

[Previous page](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247.md?page=2)