I am excited to announce a new package for data manipulation in pure Julia. Introduction InMemoryDatasets.jl is a multi-threaded package for data manipulation and is designed for Julia 1.6+ (64bit OS). The core computation engine of the package is a set of customised algorithms developed specifically for columnar tables. The package performance is tuned with two goals in mind, a) low overhead of allowing missing values everywhere, and b) the following priorities - in order of importance: Low compilation time Memory efficiency High performance I tried to keep the overall complexity of the package as low as possible to simplify: the maintenance of the package adding new features to the package contributing to the package Features InMemoryDatasets.jl has many interesting features which I here highlight some of my favourites (in no particular order): Assigning a named function to a column as its format By default, formatted values are used for operations like: displaying, sorting, grouping, joining,… Format evaluation is lazy Formats don’t change the actual values Multi-threading across the whole package Most functions in InMemoryDatasets.jl exploit all cores available to Julia by default Disabling parallel computation via passing the threads = false keyword argument to functions Powerful row-wise operations Support many common operations Specialised operations for modifying columns Customised row-wise operations for filtering observations / filter simply wraps byrow Unique approach for reshaping data Unified syntax for all type of reshaping Cover all reshaping functions: stacking and un-stacking on single/multiple columns wide to long and long to wide reshaping transposing and more Fast sorting algorithms Stable and Unstable HeapSort and QuickSort algorithms Count sort for integers Compiler friendly grouping algorithms groupby!/groupby to group observation using sorting algorithms - sorted order gatherby to group observation using hybrid hash algorithms - observations order incremental grouping operation for groupby!/groupby, i.e. adding a column at a time Efficient joining algorithms Preserve the order of observations in the left data set Support two methods for joining: sort-merge join and hash join. Customised columnar-hybrid-hash algorithms for join Inequality-kind (non-equi) and range joins for innerjoin, contains, semijoin!/semijoin, antijoin!/antijoin closejoin!/closejoin for non exact match join update!/update for updating a master data set with values from a transaction data set Benchmarks The following benchmarks present preliminary results for InMemoryDatasets.jl(IMD)'s performance compared to some data manipulation packages. The benchmarks are based on the db-benchmark repository with few changes: Except DataFrames.jl (and basic familiarity with pandas), I have very limited knowledge about other solutions listed in db-benchmark. Thus, I simply pick some of the solutions with easy setup from the top: polars, DataFrames.jl, data.table I report the total time, and use fail when a solution can’t complete a task I use a Linux REHL7 machine with 16 cores and 128GB of memory. The machine is not ideal for benchmarking (not dedicated or isolated), however, the situation is the same for all solutions and I randomise the order of runs to alleviate the problem. Anyway, I have submitted a PR to the db-benchmark project. The results are based on the latest version of the solutions + the latest PRs submitted to the db-benchmark project. I only select the advanced questions for the groupby task. The main reason is that IMD uses less rigorous approach for basic questions - priority of optimisation. Only data sets with no missing values are considered. The reason for this is that I initially included pandas and it couldn’t handle columns with missing values in groupby. Nevertheless, this has no effect on IMD, since IMD always uses Union type of Missing for columns. For the join task I increase the machine’s memory to 178GB. I add 50GB of memory to make sure that there is enough space for loading data for the 50GB case. The groupby task timing in seconds - smaller is better Numbers in parentheses are total time for the second run only (mitigating the compilation time for IMD and DF.jl) Data IMD polars DF.jl DT 1e7 - 2e0 9(3) 6(3) 17(6) 136(67) 1e7 - 1e1 8(2) 4(2) 13(5) 32(15) 1e7 - 1e2 9(3) 4(2) 10(3) 10(5) 1e8 - 2e0 60(28) 64(33) 150(73) 1152(586) 1e8 - 1e1 51(23) 48(24) 118(56) 414(198) 1e8 - 1e2 50(22) 41(20) 81(39) 106(52) 1e9 - 2e0 540(267) fail fail fail 1e9 - 1e1 514(258) fail fail 3945(1904) 1e9 - 1e2 439(221) fail fail 1143(546) The join task timing in seconds - smaller is better Data IMD polars DF.jl DT 1e7 5(1.5) 3(1.5) 8(3) 8(4) 1e8 46(22) 43(23) 91(45) 88(46) 1e9 356(178) fail fail fail Acknowledgement I like to acknowledge the contributors to Julia’s data ecosystem, especially DataFrames.jl, since the existence of their works gave the development of InMemoryDatasets.jl a head start.

It’s strange Polars uses so much memory. Isn’t Polars internally a Rust library and isn’t Rust supposed to use memory more efficiently than Julia?

Julia is generally very memory efficient. It has around 200 Mb overhead to launch, but it gives you very good tools to write memory efficient data manipulation. (The one exception is that String has unfortunately high overhead for small strings currently).

I want to point out to observers of this thread that there seems to be some funny business going on here and in other discussions related to InMemoryDatasets. Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that they may be coming from a common location/network. This does not appear to be straight up sock puppetry (a la Henning Rousseau)—posters seem to (mostly) actually be different people—but there does seem to be some sort of hidden agenda here. My guess is that the goal is to make it appear that there is more widespread dissatisfaction with DataFrames in the Julia community than there actually is. I feel I have to post this warning so that participants in the conversations are not taken in by any deception. To those people in the thread who are doing this—first of all, welcome! You mostly appear to be new to the Julia community and you come bearing code, which is great! However, please consider taking a different approach here. First, stop trying to start flame wars with DataFrames developers—that is not cool. Also, please stop trying to appear to be independent people who just happen to all be fed up with DataFrames. It’s fine if you all work together and are collectively frustrated with DataFrames. That’s absolutely ok—just don’t be deceptive about it. It is also totally fine to have developed a fork of DataFrames and compete with it. The MIT license very much allows that and if you think you can do better, by all means, give it a try. (One thing that does need to be fixed is the license copyright notice: InMemoryDatasets appears to be derived from the DataFrames code base and the MIT license does require keeping the copyright notice intact, so if you could fix that, that would put the project in legally upstanding footing.) Assuming that I’m correct about the “company that is collectively interested in improving on DataFrames” interpretation of what’s going on, my suggestion would be: take a beat, reset the conversation, be direct about working together and that you have created and are promoting InMemoryDatasets as an open source alternative to DataFrames. Maybe I’m wrong about my interpretation and if so, feel free to let me know here or privately what’s actually going on.

[image] StefanKarpinski: Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that they may be coming from a common location/network. This does not appear to be straight up sock puppetry ( a la Henning Rousseau)—posters seem to (mostly) actually be different people—but there does seem to be some sort of hidden agenda here. I think it would be good to give people a shot to give alternative explanations since I can imagine several hypothetical reasons for this beyond a desire to seem larger in size than appropriate.

Yeah, I’m absolutely open to something else being up here—feel free to DM me—but something odd is going on and it seemed like people ought to be aware.

[image] AMJ: to be done in a more subtle way The InMemoryDatasets new users have used all but “subbtle ways”

I really want to make sure that this does not become accusatory and that we don’t pile on. Nothing that has been done is terrible and it’s really exciting to have people interested enough in data wrangling in Julia to take a crack at a new package like this. It’s a lot of work and it’s generously contributed for anyone to use. Who amongst us hasn’t gotten a little vehement in our defense of Julia against its alternatives? Pointing out differences between similar packages can surely easily go the same way without ill intentions. We’re not sure what the motivation is for the funky accounts, but let’s please, please, please let’s give people the benefit of the doubt.

[image] StefanKarpinski: Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that they may be coming from a common location/network. Perhaps it’s best to leave it at that (just the facts) and avoid speculating about intentions. That way anybody from the relevant group has an opportunity to explain if they choose, without feeling defensive.

[image] jar1: Perhaps it’s best to leave it at that (just the facts) and avoid speculating about intentions. Yeah, but allow me one remark: I don’t want to read anymore posts about about the benefits of competition vs. cooperation (which we could discuss elsewhere). If we are going Darwinian in an open source forum, I’m out.

I think these kinds of posts shouldn’t be included in a public discussion, because they are more dangerous to community than any use. Including author there are less than 30 people involve in this topic and call it sizable is a little rash, I also searched topics about InMemoryDatasets and I found 7 of them so far which one of them is this announcement and two of them are also mine.

[ANN] A new lightning fast package for data manipulation in pure Julia

Package Announcements

Topic		Replies	Views
How is the data ecosystem right now for large datasets? Data	35	7351	July 13, 2017
The state of DataFrames.jl H2O benchmark Package Announcements dataframes	53	10350	January 1, 2025
Who does "better" than DataFrames? Performance dataframes	43	2823	April 6, 2023
[ANN] DataFrameDBs.jl Data package , announcement	60	5172	May 2, 2020
Julia performs poorly on group-by benchmarks Data performance	48	6669	January 23, 2019

[ANN] A new lightning fast package for data manipulation in pure Julia

Related topics