# \[ANN\] A new lightning fast package for data manipulation in pure Julia

**URL:** <https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197>\
**Category:** Package Announcements\
**Tags:** data, dataframes, inmemorydatasets\
**Created:** [March 21, 2022, 7:35am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197 "2022-03-21T07:35:42Z")\
**Posts on this page:** 20\
**Page:** 4

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 29, 2022, 3:43pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/62 "2022-03-29T15:43:23Z")

</div>

Julia is generally very memory efficient. It has around 200 Mb overhead to launch, but it gives you very good tools to write memory efficient data manipulation. (The one exception is that `String` has unfortunately high overhead for small strings currently).

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 29, 2022, 6:33pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/63 "2022-03-29T18:33:34Z")

</div>

I want to point out to observers of this thread that there seems to be some funny business going on here and in other discussions related to InMemoryDatasets. Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that they may be coming from a common location/network. This does not appear to be straight up sock puppetry (_a la_ Henning Rousseau)—posters seem to (mostly) actually be different people—but there does seem to be some sort of hidden agenda here. My guess is that the goal is to make it appear that there is more widespread dissatisfaction with DataFrames in the Julia community than there actually is. I feel I have to post this warning so that participants in the conversations are not taken in by any deception.

To those people in the thread who are doing this—first of all, welcome! You mostly appear to be new to the Julia community and you come bearing code, which is great! However, please consider taking a different approach here. First, stop trying to start flame wars with DataFrames developers—that is not cool. Also, please stop trying to appear to be independent people who just happen to all be fed up with DataFrames. It’s fine if you all work together and are collectively frustrated with DataFrames. That’s absolutely ok—just don’t be deceptive about it. It is also totally fine to have developed a fork of DataFrames and compete with it. The MIT license very much allows that and if you think you can do better, by all means, give it a try.

(One thing that does need to be fixed is the license copyright notice: InMemoryDatasets appears to be derived from the DataFrames code base and the MIT license does require keeping the copyright notice intact, so if you could fix that, that would put the project in legally upstanding footing.)

Assuming that I’m correct about the “company that is collectively interested in improving on DataFrames” interpretation of what’s going on, my suggestion would be: take a beat, reset the conversation, be direct about working together and that you have created and are promoting InMemoryDatasets as an open source alternative to DataFrames. Maybe I’m wrong about my interpretation and if so, feel free to let me know here or privately what’s actually going on.

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [March 29, 2022, 7:24pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/64 "2022-03-29T19:24:23Z")

</div>

> [@StefanKarpinski](#):
>
> Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that they may be coming from a common location/network. This does not appear to be straight up sock puppetry ( _a la_ Henning Rousseau)—posters seem to (mostly) actually be different people—but there does seem to be some sort of hidden agenda here.

I think it would be good to give people a shot to give alternative explanations since I can imagine several hypothetical reasons for this beyond a desire to seem larger in size than appropriate.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 29, 2022, 8:14pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/65 "2022-03-29T20:14:00Z")

</div>

Yeah, I’m absolutely open to something else being up here—feel free to DM me—but something odd is going on and it seemed like people ought to be aware.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 29, 2022, 8:45pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/66 "2022-03-29T20:45:59Z")

</div>

Split this topic on March 29, 2022, 8:45pm UTC

A post was split to a new topic: [Julia PR team?](https://discourse.julialang.org/t/julia-pr-team/78700)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 29, 2022, 8:48pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/67 "2022-03-29T20:48:40Z")

</div>

> [@AMJ](#):
>
> to be done in a more subtle way

The InMemoryDatasets new users have used all but “subbtle ways”

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 29, 2022, 9:07pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/68 "2022-03-29T21:07:42Z")

</div>

I really want to make sure that this does not become accusatory and that we don’t pile on. Nothing that has been done is terrible and it’s really exciting to have people interested enough in data wrangling in Julia to take a crack at a new package like this. It’s a lot of work and it’s generously contributed for anyone to use. Who amongst us hasn’t gotten a little vehement in our defense of Julia against its alternatives? Pointing out differences between similar packages can surely easily go the same way without ill intentions. We’re not sure what the motivation is for the funky accounts, but let’s please, please, please let’s give people the benefit of the doubt.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [March 29, 2022, 9:11pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/69 "2022-03-29T21:11:41Z")

</div>

> [@StefanKarpinski](#):
>
> Specifically, there appear to be a sizable number of people in these threads who are masking their location and there are indications that they may be coming from a common location/network.

Perhaps it’s best to leave it at that (just the facts) and avoid speculating about intentions. That way anybody from the relevant group has an opportunity to explain if they choose, without feeling defensive.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 29, 2022, 9:20pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/70 "2022-03-29T21:20:16Z")

</div>

> [@jar1](#):
>
> Perhaps it’s best to leave it at that (just the facts) and avoid speculating about intentions.

Yeah, but allow me one remark: I don’t want to read anymore posts about about the benefits of competition vs. cooperation (which we could discuss elsewhere). If we are going Darwinian in an open source forum, I’m out.

---

<div class="post-metadata">

**Author:** ![monopolynomial](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/monopolynomial/32/34782_2.png) [@monopolynomial](https://discourse.julialang.org/u/monopolynomial)\
**Post date:** [March 29, 2022, 10:16pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/71 "2022-03-29T22:16:25Z")

</div>

I think these kinds of posts shouldn’t be included in a public discussion, because they are more dangerous to community than any use. Including author there are less than 30 people involve in this topic and call it sizable is a little rash, I also searched topics about InMemoryDatasets and I found 7 of them so far which one of them is this announcement and two of them are also mine.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 29, 2022, 10:47pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/72 "2022-03-29T22:47:15Z")

</div>

> [@monopolynomial](#):
>
> I think these kinds of posts shouldn’t be included in a public discussion, because they are more dangerous to community than any use.

Sorry to object: I’m thankful for this kind of transparency.

On the one hand I can imagine frustrations of developers like @sl-solution to get improvement PRs rejected by mature libraries due to compatibility reasons which lead to these kinds of new developments.

On the other hand this is a bad sign, which needs to be addressed:

> [@StefanKarpinski](#):
>
> (One thing that does need to be fixed is the license copyright notice: InMemoryDatasets appears to be derived from the DataFrames code base and the MIT license does require keeping the copyright notice intact, so if you could fix that, that would put the project in legally upstanding footing.)

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 29, 2022, 10:57pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/73 "2022-03-29T22:57:35Z")

</div>

This is a very commonly misunderstood detail of the MIT license. Seems like a mistake.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [March 29, 2022, 11:12pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/74 "2022-03-29T23:12:34Z")

</div>

Yeah, that’s a very common mistake that I’ve made myself before. I think so long as people are gracious about fixing these things when brought to their attention and it’s not some clear pattern of bad behaviour, it’s best that we all assume that MIT license violations are accidental.

---

<div class="post-metadata">

**Author:** ![pursultani](https://avatars.discourse-cdn.com/v4/letter/p/977dab/32.png) [@pursultani](https://discourse.julialang.org/u/pursultani)\
**Post date:** [March 29, 2022, 11:14pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/75 "2022-03-29T23:14:45Z")

</div>

Awesome! Right package in the wrong language.

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [March 29, 2022, 11:25pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/76 "2022-03-29T23:25:17Z")

</div>

If you don’t mind: could you help me understand why you joined a Discourse forum for a language you think is bad exactly 1 minute before posting a comment about a very specific package? What was the specific chain of events that led to that happening? It seems like a remarkable coincidence.

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [March 30, 2022, 1:42am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/77 "2022-03-30T01:42:47Z")

</div>

Sincerely, the most insulting thing about this comment is not the puerile attack to the language but how it genuinely underestimates the community efficiency to spot a troll on sight.

---

<div class="post-metadata">

**Author:** ![pursultani](https://avatars.discourse-cdn.com/v4/letter/p/977dab/32.png) [@pursultani](https://discourse.julialang.org/u/pursultani)\
**Post date:** [March 30, 2022, 3:43am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/78 "2022-03-30T03:43:24Z")

</div>

This comment is intended to the original author of this post within a very specific context which I’m not obligated to share with you. Please don’t make any further assumptions.

I would appreciate it if you could point me to specific guidelines that bans such an “insult” to the language.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [March 30, 2022, 5:18am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/79 "2022-03-30T05:18:10Z")

</div>

Who said it was banned?

Calling someone out for poor behavior can be done regardless of whether that behavior is specifically banned.

And poor behavior is not ok just because you have a hidden agenda, as you yourself admit.

---

<div class="post-metadata">

**Author:** ![lawless-m](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lawless-m/32/30869_2.png) [@lawless-m](https://discourse.julialang.org/u/lawless-m)\
**Post date:** [March 30, 2022, 7:54am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/80 "2022-03-30T07:54:41Z")

</div>

> [@pursultani](#):
>
> I would appreciate it if you could point me to specific guidelines that bans such an “insult” to the language.

[https://discourse.julialang.org/guidelines](https://discourse.julialang.org/guidelines)

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 30, 2022, 8:03am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/81 "2022-03-30T08:03:33Z")

</div>

> [@pursultani](#):
>
> Awesome! Right package in the wrong language.

From the first reactions here I indeed conclude this to be a significant improvement. I’d also expect that the author put some thought into which language to use. So where do you think he misjudged?

> [@pursultani](#):
>
> This comment is intended to the original author of this post within a very specific context which I’m not obligated to share with you.

Granted. But if you are not interested in discussing this publicly you could use Discourse PM instead.

[Previous page](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197.md?page=3)

[Next page](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197.md?page=5)