# Rowwise compuation in \`InMemoryDatasets.jl\` vs \`DataFrames.jl\`

**URL:** <https://discourse.julialang.org/t/rowwise-compuation-in-inmemorydatasets-jl-vs-dataframes-jl/78314>\
**Category:** Performance\
**Tags:** data, dataframes, inmemorydatasets\
**Created:** [March 23, 2022, 12:19am UTC](https://discourse.julialang.org/t/rowwise-compuation-in-inmemorydatasets-jl-vs-dataframes-jl/78314 "2022-03-23T00:19:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![monopolynomial](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/monopolynomial/32/34782_2.png) [@monopolynomial](https://discourse.julialang.org/u/monopolynomial)\
**Post date:** [March 23, 2022, 12:19am UTC](https://discourse.julialang.org/t/rowwise-compuation-in-inmemorydatasets-jl-vs-dataframes-jl/78314/1 "2022-03-23T00:19:37Z")

</div>

`byrow` function in `InMemoryDatasets` is fantastic 👍 , I have done a small benchmark to compare its performance against `DataFrames.jl` for relatively wide tables. `argmax` in `DataFrames.jl` takes long!

# InMemoryDatasets

```julia
julia> using InMemoryDatasets

julia> using BenchmarkTools

julia> x = Dataset(rand(10^4, 1000), :auto);

julia> @btime byrow(x, sum, :);
  9.107 ms (7056 allocations: 263.27 KiB)

julia> @btime byrow(x, maximum, :);
  12.277 ms (7056 allocations: 263.27 KiB)

julia> @btime byrow(x, mean, :);
  11.602 ms (14096 allocations: 554.86 KiB)

julia> @btime byrow(x, argmax, :);
  17.819 ms (14140 allocations: 766.41 KiB)

```

# DataFrames (updated code)

```julia
julia> using DataFrames

julia> using BenchmarkTools

julia> x = DataFrame(rand(10^4, 1000), :auto);

julia> allowmissing!(x);

julia> @btime select(x, AsTable(:) => ByRow(sum));
  40.595 ms (2664 allocations: 264.89 KiB)

julia> @btime select(x, AsTable(:) => ByRow(maximum));
  40.707 ms (1665 allocations: 233.67 KiB)

julia> @btime select(x, AsTable(:) => ByRow(mean));
  45.278 ms (2665 allocations: 264.92 KiB)

julia> @btime select(x, AsTable(:) => ByRow(argmax∘collect));
  972.618 ms (21497934 allocations: 567.97 MiB)

```

I don’t know even how I should do byrow `any`, `all`, `select`, `coalesce`,… in `DataFrames.jl`.

---

<div class="post-metadata">

**Author:** ![huang\_min](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/huang_min/32/34337_2.png) [@huang\_min](https://discourse.julialang.org/u/huang_min)\
**Post date:** [March 23, 2022, 2:02am UTC](https://discourse.julialang.org/t/rowwise-compuation-in-inmemorydatasets-jl-vs-dataframes-jl/78314/2 "2022-03-23T02:02:47Z")

</div>

IMD seems to have multithreading problem in my machine. So the single threading results are here for comparision.

```julia

julia> Threads.nthreads()

8

julia> using InMemoryDatasets, BenchmarkTools

julia> x = Dataset(rand(10^4, 1000), :auto);

julia> @btime byrow(x, sum, :);

  23.465 ms (1070 allocations: 125.58 KiB)

julia> @btime byrow(x, maximum, :);

  32.853 ms (1070 allocations: 125.58 KiB)

julia> @btime byrow(x, mean, :);

  29.235 ms (2124 allocations: 279.48 KiB)

julia> @btime byrow(x, argmax, :);

  48.478 ms (2165 allocations: 397.22 KiB)

```

and

```julia

julia> using DataFrames, BenchmarkTools

julia> x = DataFrame(rand(10^4, 1000), :auto);

julia> allowmissing!(x);

julia> @btime select(x, AsTable(:) => ByRow(sum));

  26.776 ms (2664 allocations: 264.89 KiB)

julia> @btime select(x, AsTable(:) => ByRow(maximum));

  27.197 ms (1665 allocations: 233.67 KiB)

julia> @btime select(x, AsTable(:) => ByRow(mean));

  33.093 ms (2665 allocations: 264.92 KiB)

julia> @btime select(x, AsTable(:) => ByRow(argmax));

  29.952 s (34942165 allocations: 150.43 GiB)

```

and

```julia

julia> using DataFrames, BenchmarkTools

julia> x = DataFrame(rand(10^4, 1000), :auto);

julia> #allowmissing!(x);

julia> @btime select(x, AsTable(:) => ByRow(sum));
  4.033 ms (1654 allocations: 223.36 KiB)

julia> @btime select(x, AsTable(:) => ByRow(maximum));
  5.436 ms (1654 allocations: 223.36 KiB)

julia> @btime select(x, AsTable(:) => ByRow(mean));
  4.051 ms (1654 allocations: 223.36 KiB)

julia> @btime select(x, AsTable(:) => ByRow(argmax));
  26.154 s (30051676 allocations: 75.85 GiB)

```

argmax seems to have improved a lot.

---

<div class="post-metadata">

**Author:** ![DataFrames](https://avatars.discourse-cdn.com/v4/letter/d/e19b73/32.png) [@DataFrames](https://discourse.julialang.org/u/DataFrames)\
**Post date:** [March 23, 2022, 4:31am UTC](https://discourse.julialang.org/t/rowwise-compuation-in-inmemorydatasets-jl-vs-dataframes-jl/78314/3 "2022-03-23T04:31:05Z")

</div>

your `DataFrames.jl` code for `argmax` should be changed to:

```julia
julia> @btime select(x, AsTable(:) => ByRow(argmax∘collect));

```

> [@monopolynomial](#):
>
> I don’t know even how I should do byrow `any` , `all` , `select` , `coalesce` ,… in `DataFrames.jl` .

[this](https://www.juliabloggers.com/any-and-all-reduction-of-rows-in-dataframes-jl/) blog has some way to deal with `any` and `all` in `DataFrames.jl`.