# Combine with regex column selector?

**URL:** <https://discourse.julialang.org/t/combine-with-regex-column-selector/87520>\
**Category:** Data\
**Tags:** dataframes\
**Created:** [September 20, 2022, 1:27pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520 "2022-09-20T13:27:48Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2022, 1:27pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/1 "2022-09-20T13:27:48Z")

</div>

I’m a bit puzzled by this example, where I expected the same column selector to work both for `select` as for `combine`. For some reason I’m unable to use the regex selector with `combine`:

```julia
julia> using DataFrames, Statistics

julia> df = DataFrame(a=repeat([1, 2, 3, 4], outer=[2]),
                               b1=repeat([2, 1], outer=[4]),
                               b2=1:8)

julia> combine(df, [:b1, :b2] .=> mean)
1×2 DataFrame
 Row │ b1_mean b2_mean 
     │ Float64 Float64 
─────┼──────────────────
   1 │ 1.5 4.5

julia> select(df, r"b\d") # works
8×2 DataFrame
 Row │ b1 b2    
     │ Int64 Int64 
─────┼──────────────
   1 │ 2 1
   2 │ 1 2
   3 │ 2 3
   4 │ 1 4
   5 │ 2 5
   6 │ 1 6
   7 │ 2 7

julia> combine(df, r"b\d" .=> mean)
ERROR: MethodError: objects of type Vector{Int64} are not callable
Use square brackets [] for indexing an Array.

```

Moreover, kind of confusing, `AsTable` works with the regex, but I expected one row with 2 numbers as output:

```julia
julia> combine(df, AsTable(r"b\d") .=> mean)
8×1 DataFrame
 Row │ b1_b2_mean 
     │ Float64    
─────┼────────────
   1 │ 1.5
   2 │ 1.5
   3 │ 2.5
   4 │ 2.5
   5 │ 3.5
   6 │ 3.5
   7 │ 4.5
   8 │ 4.5

```

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [September 20, 2022, 1:44pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/2 "2022-09-20T13:44:19Z")

</div>

I’m more surprised that `select` works with the regex selector, I tend to go for

```julia
combine(df, names(df, r"b\d") .=> mean)

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 20, 2022, 1:52pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/3 "2022-09-20T13:52:57Z")

</div>

I’m not sure why `AsTable` works with Regex. There could be a `Base.Broadcastable` definition somewhere. But for

```julia
julia> combine(df, r"b\d" .=> mean)

```

this is correct behavior. DataFrames.jl does not get to override Base broadcasting rules for Regex. That’s why `Cols` was recently modified to allow broadcasting.

EDIT: There is indeed a `Base.Broadcastable` definition. See [here](https://github.com/JuliaData/DataFrames.jl/blob/34624f3e36e2964cf64a1a95729565669bdbf434/src/other/utils.jl#L75). Not sure the reason but I’m sure there is one.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2022, 1:54pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/4 "2022-09-20T13:54:05Z")

</div>

in any case there seems to be an inconsistency with the docs at `?select`, which says:

> All these functions take a specification of one or more functions to apply to each subset of the DataFrame. This specification can be  
> of the following forms:

```
1. standard column selectors (integers, Symbols, strings, vectors of integers, vectors of Symbols, vectors of strings, All,
   Cols, :, Between, Not and regular expressions)

```

_All these functions_ are `select`, `transform` and `combine`.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2022, 1:55pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/5 "2022-09-20T13:55:29Z")

</div>

let’s not get hung up on the `AsTable` broadcast, that may be a different issue. I was just trying all sorts of stuff.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 20, 2022, 1:55pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/6 "2022-09-20T13:55:32Z")

</div>

I don’t think this is wrong. You can do

```julia
foo(args...) = 1
select(df, r"a/b" => foo)

```

which doesn’t use broadcasting.

> [@floswald](#):
>
> All these functions take a specification of one or more functions to apply to each subset of the DataFrame. This specification can be  
> of the following forms:

This doesn’t mention broadcasting at all. And broadcasting _i think_ is the only thing making seemingly inconsistent behavior. But if working with collections and broadcasting is inconsistent within DataFrames.jl, that’s fine because its consistent with Base.

I think the philosophy on some of these docs is that Base rules take precedent. Issues like this might not need to be documented if they are downstream of Base behavior.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2022, 1:57pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/7 "2022-09-20T13:57:25Z")

</div>

simple question is: how do I do this

```julia
combine(df, [:b1, :b2] .=> mean)

```

if I have 50 b’s from 1-50?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 20, 2022, 2:00pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/8 "2022-09-20T14:00:45Z")

</div>

I would use `Cols(r"b\d") .=> mean`

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2022, 2:03pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/9 "2022-09-20T14:03:20Z")

</div>

ok yes that works. My point is: that was really hard to find out. If I see

```julia
select(df, r"b")

```

works, and I read the docs, I expect that

```julia
combine(df, r"b" .=> fun)

```

works as well. not really clear why one needs the `Cols` in one case and not the other.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 20, 2022, 2:04pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/10 "2022-09-20T14:04:54Z")

</div>

Yes, that’s definitely frustrating.

You have to know that Regex isn’t broadcastable first, then things make sense.

Maybe `Cols` should be highlighted more in the docs because it exists to solve this exact problem.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2022, 2:07pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/11 "2022-09-20T14:07:50Z")

</div>

ok i see. I somehow though that the regex just returns a vector of column names, but that’s obviously wrong - I guess that’s what `Cols` does. yeah that should be made explicit somewhere, because the regex selector is so important. I think.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 20, 2022, 2:09pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/12 "2022-09-20T14:09:47Z")

</div>

Remember that the major benefit of the `src => fun => dest` mini-language is that there is no magic, unlike `dplyr`. Everything gets evaluated as normal, meaning the `src => fun => dest` gets evaluated first, before DataFrames.jl can do anything with it.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2022, 2:12pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/13 "2022-09-20T14:12:21Z")

</div>

That _is_ actually really helpful. mayb that’s a good strategy to compose a call to start with! I’ll try to remember this.

```julia
julia> [:a , :b] .=> mean .=> [:out1, :out2]
2-element Vector{Pair{Symbol, Pair{typeof(mean), Symbol}}}:
 :a => (Statistics.mean => :out1)
 :b => (Statistics.mean => :out2)

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 20, 2022, 2:17pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/14 "2022-09-20T14:17:21Z")

</div>

Actually, I was wrong about my explanation above. Regex is broadcasted like `Ref`.

```julia

julia> r"a" .=> [1, 2]
2-element Vector{Pair{Regex, Int64}}:
 r"a" => 1
 r"a" => 2

```

What’s going on is that your call in combine creates

```julia
julia> r"b\d" .=> mean
r"b\d" => Statistics.mean

```

Notice it’s not a vector… so I would expect it’s trying to do

```julia
mean([1, 2], [3, 4], [5,6])

```

or similar. This expectation is correct, and the error I get is

```julia
julia> combine(df, r"b\d" .=> mean)
ERROR: MethodError: no method matching mean(::Vector{Float64}, ::Vector{Float64}, ::Vector{Float64}, ::Vector{Float64},

```

which is _different_ than your original error of

```julia
julia> combine(df, r"b\d" .=> mean)
ERROR: MethodError: objects of type Vector{Int64} are not callable
Use square brackets [] for indexing an Array.

```

so no, I don’t know what’s going on exactly. But the solution remains the same: Use `Cols` to broadcast Regex inside `src => fun => dest`

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 20, 2022, 4:06pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/15 "2022-09-20T16:06:53Z")

</div>

Yes it’s annoying that regexes don’t work like `Cols`, `Between` and `Not`, but there’s no solution to that as it’s defined in Base. This kind of thing could be handled more consistently in DataFramesMeta, but wrapping in `Cols` is the easiest solution.

The error @floswald gets with `combine(df, r"b\d" .=> mean)` is due to calling `mean(df.b1, df.b2)`, which interprets `df.b1` as a function to apply to each element of `df.b2`. Unfortunately the error is cryptic, and varies according to the number of columns that match the regex.

`AsTable` is broadcastable to allow things such as applying a series of functions to each row in a table efficiently. See the last example of [this section](https://dataframes.juliadata.org/stable/man/working_with_dataframes/#Selecting-and-transforming-columns) of the manual:

```julia
julia> transform(df, AsTable(:) .=>
                     ByRow.([sum∘skipmissing,
                             x -> count(!ismissing, x),
                             mean∘skipmissing]) .=>
                     [:sum, :n, :mean])

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [September 20, 2022, 4:33pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/16 "2022-09-20T16:33:35Z")

</div>

Let me summarize the issue here as there are many things into play in it:

- `Regex` can be used in normal operation specification like `r"a" => fun`, which passes columns selected by `r"a"` as positional arguments to `fun`
- in broadcasting both `Regex` and `AsTable` are scalars so `r"a" .=> fun` and `AsTable(...) .=> fun` are just the same as `r"a" => fun` and `AsTable(...) => fun`. As @nalimilan commented - this works this way for `Regex` because Base Julia defines it this way. Also `Colon()` (`:`) behaves this way - it is defined to be a scalar in broadcasting in Base Julia so writing `(:) .=> fun` is the same as `(:) => fun`
- On the other hand `Cols`, `Not`, `Between`, `All` are defined in data ecosystem and they expand the columns selected by them when broadcasted, in other words they are treated as vectors of names selected by them in broadcasting

Now the second aspect is `mean`, you need to understand its contract to see why things work this way. See it by example:

```julia
julia> mean([1,2], [3,4], [5,6]) # this is what happens if you write r"a" => mean
# Error

julia> mean((a1=[1,2], a2=[3,4], a3=[5,6])) # this is what happens if you write AsTable(r"a") => mean
2-element Vector{Float64}:
 3.0
 4.0

```

In summary:

- all is consistent and correct
- you just need to know and remember that `Regex`, `AsTable`, and `:` are scalars in broadcasting, while `All`, `Cols`, `Not`, and `Between` are vectors that contain columns selected by them.

This second special case is explained in [Split-apply-combine · DataFrames.jl](https://dataframes.juliadata.org/stable/man/split_apply_combine/) (the first case is not listed there as this is a default behavior in broadcasting of scalars in Base Julia). I copy-paste the relevant part:

> Note! If `cols` or `target_cols` are one of `All` , `Cols` , `Between` , or `Not` , broadcasting using `.=>` is supported and is equivalent to broadcasting the result of `names(df, cols)` or `names(df, target_cols)` . This behaves as if broadcasting happened after replacing the selector with selected column names within the data frame scope.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2022, 5:19pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/17 "2022-09-20T17:19:02Z")

</div>

> [@bkamins](#):
>
> `mean((a1=[1,2], a2=[3,4], a3=[5,6]))`

wow I had no idea that this even works. So much for that!

Thanks for the explanations, I think I understand most of it now. One has to know Base Julia well in order to use the minilanguage effectively. I guess I was making too quick conclusions from this

```julia
select(df, r"b")

```

to that

```julia
combine(df, r"b" .=> fun)

```

and said it’s inconsistent because it does not work this way. I guess the key mistake was to think that the second command first does something equivalent to the `select` (tease out some columns), in order to then operate on that selected dataframe - which is not wat’s happening. I think that writing out the `src => fun` part like @pdeffebach showed me is quite helpful for me.

will try to remember the golden rule (“you just need to know and remember…”) there! Thanks again! 🙂

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [September 20, 2022, 5:34pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/18 "2022-09-20T17:34:02Z")

</div>

> [@floswald](#):
>
> One has to know Base Julia well

Agreed. However, in general when using Julia it is a valid statement if you want to be sure that Julia does what you expect 😄. Broadcasting is powerful, but pretty complex if you dig deeper into it (fortunately in common cases it is easy).

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [September 20, 2022, 6:01pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/19 "2022-09-20T18:01:44Z")

</div>

After this really enlightening discussion I feel vindicated for using `names(df, r"... ")` which transparently creates a simple vector of strings that even I can reason about 😀

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [September 20, 2022, 8:06pm UTC](https://discourse.julialang.org/t/combine-with-regex-column-selector/87520/20 "2022-09-20T20:06:07Z")

</div>

I would like to add a topic, for possible further study, to this interesting discussion.  
Instead of acting on the input to the left of =\> you can act on the right part (the function func) in the following ways, obtaining the average by rows or by columns.

```julia
combine(df, r"b\d" => (x...)->mean(x))

combine(df, r"b\d" => ((x...)->[mean.(x)...;;])=>AsTable)
combine(df, r"b\d" => ((x...)->[mean.(x)...]')=>AsTable)

```

which in the case of two columns can be written like this

```julia
combine(df, r"b\d" => (x,y)->mean((x,y)))
combine(df, r"b\d" => ((x,y)->mean.([x,y])')=>AsTable)

```