# Future directions for DataFrames.jl

**URL:** <https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247>\
**Category:** Data\
**Tags:** package, dataframes\
**Created:** [August 12, 2021, 6:23am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247 "2021-08-12T06:23:11Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [August 12, 2021, 5:52pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/21 "2021-08-12T17:52:48Z")

</div>

Jumping in here to strongly second the point about Dask in particular. As simple and elegant as the partitioning model is in theory, in practice I’ve found Dask’s implementation to be an extremely leaky abstraction. Stare at it funny, and the runtime will decide to run everything serially or materialize all of your data anyways. Hopefully DTable does not experience the same pathologies, but I’ll have to claim ignorance there 🙂

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [August 12, 2021, 5:56pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/22 "2021-08-12T17:56:25Z")

</div>

Pinned on August 12, 2021, 5:56pm UTC



---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 12, 2021, 8:00pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/23 "2021-08-12T20:00:48Z")

</div>

> [@viralbshah](#):
>
> One question is, whether using abstractions like KernelAbstractions or Dagger, we can target more hardware configurations with the same codebase?

This is essentially what I discussed in [[ANN] Folds.jl: threaded, distributed, and GPU-based high-level data-parallel interface for Julia](https://discourse.julialang.org/t/ann-folds-jl-threaded-distributed-and-gpu-based-high-level-data-parallel-interface-for-julia/54701) and

[![](https://global.discourse-cdn.com/julialang/original/3X/6/2/6254dfcced4fd4d5737004d06c5e6aafc91f7e82.jpeg "JuliaFolds: Structured parallelism for Julia | Takafumi Arakaki | JuliaCon2021") ](https://www.youtube.com/watch?v=TodjM8gCylI)

KernelAbstractions and Dagger are great in that they are very flexible and generic. But having more higher-level _structured_ representation helps for lowering to target-specific programs. As always, adding more constraints can help for encoding structures of your code which then can be exploited by the underlying framework for performance and composability.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 12, 2021, 11:43pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/24 "2021-08-12T23:43:48Z")

</div>

> [@viralbshah](#):
>
> will the I/O cost make it practically intractable?

I think scaling to about 10x is still doable. Especially with new tech like 3D XPoint drives becoming more common. For example, I have played with the Fannie Mae data (200G in CSV format) and can manipulate it using {disk.frame} in R which works on the principle of splitting up the data into smaller files and using multiple process to process the same file.

Beyond that size, you are probably better off with a cluster based approach, but cluster based should be a last resort. Most companies don’t need a spark but they end up paying the spark tax just so they can say “big data”.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 12, 2021, 11:48pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/25 "2021-08-12T23:48:00Z")

</div>

> [@pdeffebach](#):
>
> To clarify, I don’t think the proposal is to put out-of-core things _into_ DataFrames. Ideally, DataFrames would formalize the assumptions it has on vectors and the APIs it uses with vectors internally so that distributed arrays will “just work”.

Yeah. I had done a quick study of the possibility of doing this with `groupby`. There’s a part of the `groupby` code that materialized an index in memory. This may not fit well with out-of-core datasets. See my [research notes](https://github.com/xiaodaigh/DiskFrames.jl/blob/main/research/notes.md).

So for `groupby` to do that it might need to make that part of the code generic, which can add pressures to readability and maintainability. For now, I think it’s a case of let hundred flowers boom and let people try out multiple approaches. I am keen to try `Dagger.DTable` soon.

---

<div class="post-metadata">

**Author:** ![tbenst](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbenst/32/3029_2.png) [@tbenst](https://discourse.julialang.org/u/tbenst)\
**Post date:** [August 13, 2021, 12:16am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/26 "2021-08-13T00:16:10Z")

</div>

Huge fan of DataFrames.jl—I think it’s much more ergonomic than eg Pandas.

I sometimes see [Database-like ops benchmark](https://h2oai.github.io/db-benchmark/) mentioned when folks compare R and Julia, and it seems like in most cases DF.jl stacks up well, but does OOM more frequently on certain benchmarks. Perhaps focusing on memory footprint would be a useful intermediate step towards supporting larger and larger data sizes?

Also interesting to see how poorly Dask does from a performance standpoint

---

<div class="post-metadata">

**Author:** ![blackeneth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/blackeneth/32/10353_2.png) [@blackeneth](https://discourse.julialang.org/u/blackeneth)\
**Post date:** [August 13, 2021, 2:59am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/27 "2021-08-13T02:59:06Z")

</div>

That PR, though, bounces you over to [What metadata should be #2276](https://github.com/JuliaData/DataFrames.jl/issues/2276), which then bounces you over to [DataAPI metadata method #22](https://github.com/JuliaData/DataAPI.jl/issues/22)

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 13, 2021, 6:08am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/28 "2021-08-13T06:08:24Z")

</div>

> [@blackeneth](#):
>
> That PR, though, bounces you over

And this is the issue - the problem is complex on the API level. I have agreed with @pdeffebach to try to come up with some clean proposal this year.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 13, 2021, 6:11am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/29 "2021-08-13T06:11:25Z")

</div>

> [@tbenst](#):
>
> but does OOM more frequently on certain benchmarks.

[https://github.com/h2oai/db-benchmark/pull/232](https://github.com/h2oai/db-benchmark/pull/232) should fix this at least partially and in the future what this PR does will be done automatically. We just need to wait for benchmark maintainers to merge it and run the benchmarks to see the impact.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 13, 2021, 7:54am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/30 "2021-08-13T07:54:18Z")

</div>

> [@nilshg](#):
>
> I wonder if you could comment on what the multi threading option entails? Is it about multi threading more internals for performance gain or better support for users to multithread their analysis?

[Here](https://bkamins.github.io/julialang/2021/08/13/threading.html) I have summarized the state of multi-threading support in DataFrames.jl.

Further work on multi-threading will extend the range of operations in which it is used automatically.

Manual multi-threading is already fully supported as internally we only use task-based multi-threading constructs. Therefore users should just use the standard functionality of Julia to perform manual multi-threading.

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 13, 2021, 8:22am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/31 "2021-08-13T08:22:31Z")

</div>

My priority:

- **Make It Work = handling data larger than fits into RAM**
- Make It Right
- Make It Fast = improve current support of multi-threading ; GPU support

### [Make It Work Make It Right Make It Fast](https://proxy.c2.com/cgi/fullSearch?search=MakeItWorkMakeItRightMakeItFast)

> This formulation of this statement has been attributed to [KentBeck](https://wiki.c2.com/?KentBeck); it has existed as part of the [UnixWay](https://wiki.c2.com/?UnixWay) for a long time.See [ButlerLampson](https://wiki.c2.com/?ButlerLampson)’s “Hints for Computer System Design” (1983) [Home - Microsoft Research](http://research.microsoft.com/en-us/um/people/blampson/33-hints/webpage.html) and Stephen C. Johnson and Brian W. Kernighan’s “The C Language and Models for Systems Programming” in Byte magazine (August 1983) ( **“the strategy is definitely: first make it work, then make it right, and, finally, make it fast.”**  
> )  
> …

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 13, 2021, 8:35am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/32 "2021-08-13T08:35:52Z")

</div>

> [@xiaodai](#):
>
> 3D XPoint drives

> On March 16, 2021, Micron announced that it would cease development of 3D XPoint in order to develop products based on Compute Express Link (CXL)

seems like a rapidly changing field and idk if “your local cluster” will have these SSDs…

---

<div class="post-metadata">

**Author:** ![bigdataman2015](https://avatars.discourse-cdn.com/v4/letter/b/e99b99/32.png) [@bigdataman2015](https://discourse.julialang.org/u/bigdataman2015)\
**Post date:** [August 13, 2021, 10:04am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/33 "2021-08-13T10:04:46Z")

</div>

dplyr + tidyr + purrr

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 13, 2021, 10:51am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/34 "2021-08-13T10:51:55Z")

</div>

> [@jling](#):
>
> Compute Express Link (CXL)

woah… there hasn’t been wide adoption of 3D XPoint yet. Gonna read up on CXL.

But faster disks just means better out-of-core performance. Big memory is eating big data.

Thanks

But [Intel might still be in 3D XPoint](https://www.electropages.com/blog/2021/04/micron-abandoning-3d-xpoint-favour-cxl#:~:text=Recently%2C%20Micron%20announced%20that%20it,memory%20in%20favour%20of%20CXL.) though

 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/f/8f42f246aef3fe485b18a76be8c299f449407fff.png)

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [August 13, 2021, 11:49am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/35 "2021-08-13T11:49:14Z")

</div>

That’s a good benchmark, current of almost for Julia’s DF, but I found for out-of-core: [Benchmarking disk.frame 2: disk.frame beats Dask again! disk.frame beats JuliaDB again! Anyone else wanna challenge? • disk.frame](https://diskframe.com/articles/vs-dask-juliadb-2.html)

I didn’t notice a time-stamp, so I’m not sure if it’s current. Note, Dagger.jl is a dependency of JuliaDB, and is that currently the best out-of-core support? Which I do not need, currently…

---

<div class="post-metadata">

**Author:** ![tbenst](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbenst/32/3029_2.png) [@tbenst](https://discourse.julialang.org/u/tbenst)\
**Post date:** [August 15, 2021, 8:04pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/36 "2021-08-15T20:04:28Z")

</div>

For under $1000, you can RAID0 4x nvme drives…I have a 4TB array with sustained 7-8GB/s (ie disk filling) read/write using Samsung 970 Pro (note that many [more recent drives typically optimize for consumer workflows of high burst but low sustained](https://www.anandtech.com/show/16087/the-samsung-980-pro-pcie-4-ssd-review/3)).

Top of the line interconnects are 200Gb/s (25GB/s), and that’s theoretical max, and usually shared, so I’d be surprised if a cluster approach would be faster until jobs are compute bound.

Routinely read 100GB of data into ram in about 15 seconds. Personally, if I would probably choose a beefy machine over cluster for data up to ~5TB, but depends on specific computation workflow of course.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [August 15, 2021, 10:38pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/37 "2021-08-15T22:38:37Z")

</div>

I think there’s still room for improvement in the syntax and ergonomics. It’s quite clunky to do many basic data transformations compared to something like kdb+/q.  
It would also be really nice to have type stability so that one can use dataframes anywhere without worrying about a huge drop in performance.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [August 15, 2021, 10:43pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/38 "2021-08-15T22:43:01Z")

</div>

[DataFramesMeta.jl](https://github.com/JuliaData/DataFramesMeta.jl) provides macros that construct `src => fun => dest` transformations with a more convenient syntax. It’s maintained by the JuliaData collaborators, so it has a high bus factor.

Take a look at the tutorial [here](https://juliadata.github.io/DataFramesMeta.jl/stable/dplyr/).

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [August 16, 2021, 12:43am UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/39 "2021-08-16T00:43:52Z")

</div>

Not having GPU or multi-threading support will just make your computation slower. Being unable to handle larger datasets will completely prevent you from computing anything.  
Then it’s much more important to have an out-of-core functionality.

Anyway, we don’t just need a sort of SQL database system to perform simple read and write operations, we need to be able to apply any function and library to that data, even if intermediate operations need more memory than RAM.

---

<div class="post-metadata">

**Author:** ![Thibaut](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thibaut/32/10056_2.png) [@Thibaut](https://discourse.julialang.org/u/Thibaut)\
**Post date:** [August 18, 2021, 8:14pm UTC](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247/40 "2021-08-18T20:14:59Z")

</div>

Wow, this is hard… Let’s go one per one…

**GPU support**

It could be quite nice to have faster calculations when doing dataframe operations. However, as for myself it is rare that the calculations related to data-frame manipulations are the bottleneck when working with in-RAM datasets. It is more generally the predictive models. Especially the training of experiments during the design phases.

**Improve current support of multi-threading**

Similar story, while I believe that distributing the operations is an important topic for large datasets.

**Handling data larger than fits into RAM**

For me, this would be the first thing to do. Then we will need the above performance improvements because of that…

Also, I believe that Julia has what it takes to become a very good Big Data language. It is faster than Java and Scala, easier to write. Even compared to Dask & Cie, it makes tools easier to develop and is more consistently performant since it does not need C/C++ under the hood. If Julia can build a strong ecosystem around big data it will have serious user advantages.

[Previous page](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247.md?page=1)

[Next page](https://discourse.julialang.org/t/future-directions-for-dataframes-jl/66247.md?page=3)