# \[ANN\] A new lightning fast package for data manipulation in pure Julia
**URL:** https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197
**Category:** Package Announcements
**Tags:** data, dataframes, inmemorydatasets
**Created:** [March 21, 2022, 7:35am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197 "2022-03-21T07:35:42Z")
**Posts on this page:** 16
**Page:** 5
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [March 30, 2022, 10:31am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/82 "2022-03-30T10:31:40Z")
Haven’t you considered adding the radix sort algorithm?
From discussion in other libraries (such as data.table and dataframes.jl) I’ve seen it can be faster.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [March 30, 2022, 10:50am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/83 "2022-03-30T10:50:53Z")
> [@rocco\_sprmnt21](#):
>
> In the documentation of the groupby function it says that you can use all the kwargs of the sort function and so I thought I could also use; by which is a kwarg of the Basic sort function though.
Usually in the `IMD`’s manual, for functions with similar name as in Base Julia, if I am referring to functions from Base I include `Base`, otherwise I am referring to `IMD`’s functions. In `groupby` case, I listed the accepted keywords arguments right after the sentence you are mentioning, but I think it might be better to add a link for clarification.
> [@rocco\_sprmnt21](#):
>
> I’d still rather have a result like this
> … rather than like this
Unfortunately, I am not very clear about what format you are looking for here, because those two examples are very different. Would you please elaborate on this a little more?
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [March 30, 2022, 10:58am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/84 "2022-03-30T10:58:09Z")
`IMD` always use the Union Type of `missing` and `T` for data sets’ columns, and my expectation was that those extra exported functions that you are mentioning would be used in the context of data manipulation. However, I think it would be better to make those extra functions more general.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [March 30, 2022, 11:04am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/85 "2022-03-30T11:04:46Z")
You are right.
For huge data sets, using the `String` type for a column is a dead end. I think for `IMD` we need an efficient and flexible fixed length `String` type. There is an issue about this on `IMD`’s github.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [March 30, 2022, 11:06am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/86 "2022-03-30T11:06:22Z")
> [@StefanKarpinski](#):
>
> One thing that does need to be fixed is the license copyright notice
You can track/contribute this issue on [github](https://github.com/sl-solution/InMemoryDatasets.jl/issues/45).
Fixed on master.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [March 30, 2022, 11:14am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/87 "2022-03-30T11:14:01Z")
> [@Juan](#):
>
> Haven’t you considered adding the radix sort algorithm?
[Yes](https://github.com/sl-solution/InMemoryDatasets.jl/issues/47), however, I couldn’t find any implementation of radix sort in Julia that fits to my needs.
Although the sort operations (e.g. the `groupby` function) in `IMD` are very fast, there is room for improvement.
---
### Author:  [@qsong](https://discourse.julialang.org/u/qsong)
#### Post date: [March 30, 2022, 11:52am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/88 "2022-03-30T11:52:49Z")
I was wondering if you would write a paper/blog about the sort operations in `IMD` as reading the source is not the most efficient way for users who would like to learn/understand this new package. (I found introduction blogs by either **Hadley** (for `dplyr` years ago) or @bkamins in the past two years are very useful, just to mention a few.)
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [March 30, 2022, 12:06pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/89 "2022-03-30T12:06:26Z")
> [@qsong](#):
>
> I was wondering if you would write a paper/blog about the sort operations in `IMD` as reading the source is not the most efficient way for users who would like to learn/understand this new package.
I will do this in due course.
---
### Author:  [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)
#### Post date: [March 30, 2022, 1:16pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/90 "2022-03-30T13:16:52Z")
For the expected format part I would like to have similar output if (some of) the elements of a table column are tables themselves.
This example resulted from reading several CSV files whose names were in the columns of a table. Then it was asked to add a column with the corresponding contents of the csv files.
```julia
df = DataFrame(a=1:3)
df1=DataFrame(b=11:15)
df2=DataFrame(f=["f1","f2","f3"],oc=1:3,sdf=[df,df,df1])
3×3 DataFrame
Row │ f oc sdf
│ String Int64 DataFrame
─────┼──────────────────────────────
1 │ f1 1 3×1 DataFrame
2 │ f2 2 3×1 DataFrame
3 │ f3 3 5×1 DataFrame
julia>
```
For the question of the representation of the result of a groupby done on a formatted column, I observed that if a kwarg like by of the Base.sort function was used, the resulting key column would have the original data and not the formatted ones.
The advantage, in my opinion, would be to have a clearer view of the starting situation.
One effect of applying a formatting function that is not invertible is to map two different values to the same value.
For example, in the abs case, when I see a sequence of 4 I don’t know which of these was +4 or -4.
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [March 31, 2022, 9:53am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/91 "2022-03-31T09:53:29Z")
> [@rocco\_sprmnt21](#):
>
> For the expected format part I would like to have similar output if (some of) the elements of a table column are tables themselves.
I see. Fixed it in `master`.
> [@rocco\_sprmnt21](#):
>
> For the question of the representation of the result of a groupby done on a formatted column, I observed that if a kwarg like by of the Base.sort function was used, the resulting key column would have the original data and not the formatted ones.
You can use `removeformat!` to remove unwanted formats. Setting formats to columns doesn’t change the actual values, moreover, setting and removing formats are instantaneous.
---
### Author:  [@deburko2](https://discourse.julialang.org/u/deburko2)
#### Post date: [April 1, 2022, 11:48pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/92 "2022-04-01T23:48:24Z")
> [@Juan](#):
>
> It’s strange Polars uses so much memory.
I read somewhere that `polars` needs 4times size of data memory to work smoothly.
---
### Author:  [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)
#### Post date: [June 28, 2022, 9:45pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/93 "2022-06-28T21:45:09Z")
I find it suspicious that a one-person project can provide more features and better performance than DataFrames.jl and CSV.jl. Have you submitted your benchmarks at [Database-like ops benchmark](https://h2oai.github.io/db-benchmark/)?
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [June 28, 2022, 11:18pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/94 "2022-06-28T23:18:14Z")
> [@Yifan\_Liu](#):
>
> I find it suspicious that a one-person project can provide more features and better performance than DataFrames.jl and CSV.jl
I have nothing to do with either package, and I find that somewhat inelegant. All packages are free to use and test, you are free to test whatever you find suspicious. Also it seem clear that the packages have different development constraints, so it is always possible that a new package that aims a different subset of functionalities or is not constrained by some compatibility goals can achieve better results in specific cases.
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [June 29, 2022, 1:46am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/95 "2022-06-29T01:46:02Z")
Those benchmarks are no maintained anymore. The runtime is very long.
Several users have sent them improved versions of Julia codes but the site doesn’t update the benchmarks.
---
### Author:  [@palday](https://discourse.julialang.org/u/palday)
#### Post date: [June 30, 2022, 9:16pm UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/96 "2022-06-30T21:16:52Z")
> [@monopolynomial](#):
>
> frequently need to pivot\_long\_to\_wide or visa versa but nothing was available in dataframes.
What about [`stack`/`unstack`](https://dataframes.juliadata.org/stable/lib/functions/#Reshaping-data-frames-between-tall-and-wide-formats) ?
---
### Author:  [@sl-solution](https://discourse.julialang.org/u/sl-solution)
#### Post date: [July 4, 2022, 5:31am UTC](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197/97 "2022-07-04T05:31:40Z")
> [@Yifan\_Liu](#):
>
> Have you submitted your benchmarks at [Database-like ops benchmark](https://h2oai.github.io/db-benchmark/)?
Yes.
[Previous page](https://discourse.julialang.org/t/ann-a-new-lightning-fast-package-for-data-manipulation-in-pure-julia/78197.md?page=4)