Hi, Currently DataFrames.jl is designed for in-memory processing of data frames using CPU. While still we are working on improvements (the coming year) in this area I would like to make a pool on which general design direction we should take in the long term (several years): poll I have made the p…

[image] tbenst: but does OOM more frequently on certain benchmarks. https://github.com/h2oai/db-benchmark/pull/232 should fix this at least partially and in the future what this PR does will be done automatically. We just need to wait for benchmark maintainers to merge it and run the benchmar…

[image] nilshg: I wonder if you could comment on what the multi threading option entails? Is it about multi threading more internals for performance gain or better support for users to multithread their analysis? Here I have summarized the state of multi-threading support in DataFrames.jl. F…

My priority: Make It Work = handling data larger than fits into RAM Make It Right Make It Fast = improve current support of multi-threading ; GPU support Make It Work Make It Right Make It Fast This formulation of this statement has been attributed to KentBeck ; it has existed as part of the Unix…

[image] xiaodai: 3D XPoint drives On March 16, 2021, Micron announced that it would cease development of 3D XPoint in order to develop products based on Compute Express Link (CXL) seems like a rapidly changing field and idk if “your local cluster” will have these SSDs…

[image] jling: Compute Express Link (CXL) woah… there hasn’t been wide adoption of 3D XPoint yet. Gonna read up on CXL. But faster disks just means better out-of-core performance. Big memory is eating big data. Thanks But Intel might still be in 3D XPoint though [image]

That’s a good benchmark, current of almost for Julia’s DF, but I found for out-of-core: Benchmarking disk.frame 2: disk.frame beats Dask again! disk.frame beats JuliaDB again! Anyone else wanna challenge? • disk.frame I didn’t notice a time-stamp, so I’m not sure if it’s current. Note, Dagger.jl is…

For under $1000, you can RAID0 4x nvme drives…I have a 4TB array with sustained 7-8GB/s (ie disk filling) read/write using Samsung 970 Pro (note that many more recent drives typically optimize for consumer workflows of high burst but low sustained ). Top of the line interconnects are 200Gb/s (25GB/s…

I think there’s still room for improvement in the syntax and ergonomics. It’s quite clunky to do many basic data transformations compared to something like kdb+/q. It would also be really nice to have type stability so that one can use dataframes anywhere without worrying about a huge drop in perf…

DataFramesMeta.jl provides macros that construct src => fun => dest transformations with a more convenient syntax. It’s maintained by the JuliaData collaborators, so it has a high bus factor. Take a look at the tutorial here .

Future directions for DataFrames.jl

Specific Domains Data

rafael.guerra August 19, 2021, 7:15am 41

What do you think about the solution proposed by @gcalderone? Linking also his post.

1 Like

Topic		Replies	Views
DataFrames.jl development survey Data question , dataframes	52	3329	September 27, 2020
A serious data start-up structured around a Julia data manipulation framework for larger-than-RAM data Offtopic	24	1053	September 16, 2024
The state of DataFrames.jl H2O benchmark Package Announcements dataframes	53	9786	January 1, 2025
Hierarchical or multi-index for data frames Data	10	7525	October 9, 2019
DataTables or DataFrames? Data question	32	15637	November 19, 2018

Future directions for DataFrames.jl

Related topics