# How to efficiently apply user-defined functions against grouped dataframes in \`DataFrames.jl\`?

**URL:** <https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963>\
**Category:** New to Julia\
**Tags:** dataframes\
**Created:** [February 9, 2024, 3:57am UTC](https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963 "2024-02-09T03:57:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon56682678](https://avatars.discourse-cdn.com/v4/letter/a/f475e1/32.png) [@anon56682678](https://discourse.julialang.org/u/anon56682678)\
**Post date:** [February 9, 2024, 3:57am UTC](https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963/1 "2024-02-09T03:57:56Z")

</div>

I am translating about 6k lines of R to Julia for the sake of speed and everything has been faster, significantly faster in Julia except for this specific task. I suspect what I have done below may not be the right way to apply this logic because this specific logic is easily 10-12 times faster than Julia in the old R-lang.

# Main question

What is the right way to apply user defined functions to be used inside of `filter` against grouped data frames in `DataFrames.jl`?

# Task

## Relevant objects

1. Search\_space - A vector (Have attached sample data below)
2. Gdf = A grouped dataframe (Have attached sample data below)

## Pseudocode

1. Take a grouped dataframe
2. Filter for groups where all the elements of column `col_two` is in search\_space

I have tried the following combinations

```julia
function user_defined_bool_design(my_table::SubDataFrame, search_space::Array{String, 1})

    if all(x -> in(x, search_space), my_table.col_two)
        return true
    else
        return false
    end
    
end

@time begin
    filter(sdf -> user_defined_bool_design(sdf,search_space),Gdf,ungroup=true)
end

```

The outcome of the above

_5.438332 seconds (95.41 k allocations: 12.553 MiB, 1.09% compilation time)_

```julia
function waveform_filter_list_based(my_table::GroupedDataFrame, search_space::Array{String, 1})

    hold = SubDataFrame[]

    for group in my_table

        if all(x -> in(x, search_space), group.col_two)
            push!(hold, group)
        end 
    end

    return vcat(hold...)
end

@time begin
    waveform_filter_list_based(Gdf,search_space)
end

```

The outcome of the above

_5.353175 seconds (50.79 k allocations: 4.650 MiB)_

The sample data that I have attached is significantly scaled down so the numbers may not be the same. I am working on a linux machine with 16 threads and 32 gigs of RAM.

Here is a sample search\_space

```julia-auto
search_space = [
    6.6_1,6.5_1,12.6_1,12.5_1,3.8_1,12.3_1,12.7_1,20.9_1,12.2_1,12.4_1,21.4_1,12.8_1,11.9_1,12.0_1,21.8_1,12.1_1,12.9_1,20.8_1,21.3_1,8.9_1,21.9_1,21.5_1,21.7_1,22.1_1,21.6_1,22.0_1,11.8_1,6.7_1,13.0_1,21.1_1,21.0_1,10.2_1,13.1_1,21.2_1,13.2_1,18.1_1,20.7_1,8.1_1,22.2_1,8.6_1,11.7_1,8.5_1,8.0_1,10.5_1,22.4_1,10.6_1,3.8_2,22.1_2,21.1_2,8.7_2,10.3_2,12.7_2,13.1_2,11.9_2,21.2_2,8.8_2,20.8_2,11.7_2,10.1_2,20.7_2,21.4_2,11.6_2,12.0_2,21.6_2,21.0_2,21.7_2,20.6_2,10.2_2,8.1_2,22.0_2,21.3_2,10.4_2,10.0_2,1.4_2,21.8_2,20.9_2,11.5_2,13.2_2,21.5_2,22.2_2,15.0_2,1.2_2,13.3_2,22.3_2,15.4_2,17.9_2,8.0_2,8.5_2,10.7_2,13.4_2,22.9_2,21.9_2,10.5_2,15.7_2,18.0_2,22.5_2,15.5_2,18.2_2,15.6_2,8.9_2
]

```

Here is a sample table on pastebin: [sample\_dataframe.tsv - Pastebin.com](https://pastebin.com/r4MpMsk5)

Please groupby col\_one and work with col\_two.

Is `filter()` just generally slower?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 9, 2024, 5:50am UTC](https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963/2 "2024-02-09T05:50:06Z")

</div>

Have you tried making `search_space` a `Set`?

---

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [February 9, 2024, 6:24am UTC](https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963/3 "2024-02-09T06:24:24Z")

</div>

> [@anon56682678](#):
>
> ```julia
> @time begin
> filter(sdf -> user_defined_bool_design(sdf,search_space),Gdf,ungroup=true)
> end
> 
> ```

Note that this causes repeated access to the global `search_space` and thus [is slow](https://docs.julialang.org/en/v1/manual/performance-tips/#Avoid-untyped-global-variables). That’s why you see so many allocations.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 9, 2024, 11:32am UTC](https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963/4 "2024-02-09T11:32:58Z")

</div>

Can you please report the timing of:

```julia
function f(Gdf, search_space)
    sss = Set(search_space)
    [all(x -> in(x, sss), group.col_two) for group in Gdf]
end
@time condition = f(Gdf, search_space)
@time Gdf[condition]

```

as on your data it is hard to benchmark since it is too small (I need to understand if the bottleneck is checking the condition or subsetting of `Gdf`).

---

<div class="post-metadata">

**Author:** ![anon56682678](https://avatars.discourse-cdn.com/v4/letter/a/f475e1/32.png) [@anon56682678](https://discourse.julialang.org/u/anon56682678)\
**Post date:** [February 9, 2024, 3:36pm UTC](https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963/5 "2024-02-09T15:36:07Z")

</div>

The outcome of this was

```julia
0.134730 seconds (211.57 k allocations: 19.855 MiB, 91.86% compilation time)
0.017394 seconds (16.49 k allocations: 3.565 MiB, 95.68% compilation time)

```

FYI, you might be interested in this

just trivially doing this:

```julia
function user_defined_function_set_based(my_table::SubDataFrame, search_space::Set{String})

    if all(x -> in(x, search_space), my_table.col_two)
        return true
    else
        return false
    end
    
end

```

and then

```julia
@time begin 
    filter(sdf -> user_defined_function_set_based(sdf,search_space_set),Gdf,ungroup=true)
end

```

is ` 0.021326 seconds (61.83 k allocations: 10.356 MiB, 74.50% compilation time)`. This is on data of the same size.

---

<div class="post-metadata">

**Author:** ![anon56682678](https://avatars.discourse-cdn.com/v4/letter/a/f475e1/32.png) [@anon56682678](https://discourse.julialang.org/u/anon56682678)\
**Post date:** [February 9, 2024, 3:36pm UTC](https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963/6 "2024-02-09T15:36:56Z")

</div>

Thank you! That was it. Changing list operations to set operations made things orders of magnitude faster!

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 10, 2024, 11:46am UTC](https://discourse.julialang.org/t/how-to-efficiently-apply-user-defined-functions-against-grouped-dataframes-in-dataframes-jl/109963/7 "2024-02-10T11:46:59Z")

</div>

> [@anon56682678](#):
>
> The outcome of this was

Yes - the timing confirmed that the cost was not in DataFrames.jl but lookup cost.