I am excited to announce a new package for data manipulation in pure Julia. Introduction InMemoryDatasets.jl is a multi-threaded package for data manipulation and is designed for Julia 1.6+ (64bit OS). The core computation engine of the package is a set of customised algorithms developed specifical…

It’s strange Polars uses so much memory. Isn’t Polars internally a Rust library and isn’t Rust supposed to use memory more efficiently than Julia?

Julia is generally very memory efficient. It has around 200 Mb overhead to launch, but it gives you very good tools to write memory efficient data manipulation. (The one exception is that String has unfortunately high overhead for small strings currently).

I want to point out to observers of this thread that there seems to be some funny business going on here and in other discussions related to InMemoryDatasets. Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that th…

[image] StefanKarpinski: Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that they may be coming from a common location/network. This does not appear to be straight up sock puppetry ( a la Henning Rousseau)—pos…

Yeah, I’m absolutely open to something else being up here—feel free to DM me—but something odd is going on and it seemed like people ought to be aware.

[image] AMJ: to be done in a more subtle way The InMemoryDatasets new users have used all but “subbtle ways”

I really want to make sure that this does not become accusatory and that we don’t pile on. Nothing that has been done is terrible and it’s really exciting to have people interested enough in data wrangling in Julia to take a crack at a new package like this. It’s a lot of work and it’s generously co…

[image] StefanKarpinski: Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that they may be coming from a common location/network. Perhaps it’s best to leave it at that (just the facts) and avoid speculating a…

[image] jar1: Perhaps it’s best to leave it at that (just the facts) and avoid speculating about intentions. Yeah, but allow me one remark: I don’t want to read anymore posts about about the benefits of competition vs. cooperation (which we could discuss elsewhere). If we are going Darwinian …

I think these kinds of posts shouldn’t be included in a public discussion, because they are more dangerous to community than any use. Including author there are less than 30 people involve in this topic and call it sizable is a little rash, I also searched topics about InMemoryDatasets and I found 7…

[ANN] A new lightning fast package for data manipulation in pure Julia

Package Announcements

Topic		Replies	Views
How is the data ecosystem right now for large datasets? Data	35	6986	July 13, 2017
The state of DataFrames.jl H2O benchmark Package Announcements dataframes	53	9800	January 1, 2025
Who does "better" than DataFrames? Performance dataframes	43	2302	April 6, 2023
[ANN] DataFrameDBs.jl Data package , announcement	60	4380	May 2, 2020
Julia performs poorly on group-by benchmarks Data performance	48	6046	January 23, 2019

[ANN] A new lightning fast package for data manipulation in pure Julia

Related topics