# Map over combinations of parameters, and grouping results as DataFrame

**URL:** <https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306>\
**Category:** General Usage\
**Tags:** dataframes\
**Created:** [March 2, 2022, 2:27pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306 "2022-03-02T14:27:25Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![baptnz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baptnz/32/24519_2.png) [@baptnz](https://discourse.julialang.org/u/baptnz)\
**Post date:** [March 2, 2022, 2:27pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/1 "2022-03-02T14:27:25Z")

</div>

Coming from the R world, I’m wondering how to translate the following task in a more Julian way. I often need to run a function, say `my_model(x, a, b, c)` for various combinations of parameters a,b,c, where the output of `my_model` will typically be a DataFrame with length(x) rows. Here’s a R version of this workflow,

```julia
library(purrr)
library(dplyr)
library(tidyr)

my_model <- function(x=seq(0,10, length=100), a=1, b=1, c=1, fun = cos){

  # dummy example here
  data.frame(x=x, y = a*sin(b*x) + c, z = a*fun(b*x) + c)
  
}

head(my_model())
# x y z
# 1 0.0000000 1.000000 2.000000
# 2 0.1010101 1.100838 1.994903
# 3 0.2020202 1.200649 1.979663
# 4 0.3030303 1.298414 1.954437
# 5 0.4040404 1.393137 1.919480
# 6 0.5050505 1.483852 1.875150

params <- expand.grid(a=c(0.1,0.2,0.3), b = c(1,2,3), c = c(0,0.5))
head(params)
# a b c
# 1 0.1 1 0
# 2 0.2 1 0
# 3 0.3 1 0
# 4 0.1 2 0
# 5 0.2 2 0
# 6 0.3 2 0

all <- pmap_df(params, my_model, fun = tanh, .id = 'id')
str(all)
# 'data.frame':	1800 obs. of 4 variables:
# $ id: chr "1" "1" "1" "1" ...
# $ x: num 0 0.101 0.202 0.303 0.404 ...
# $ y: num 0 0.0101 0.0201 0.0298 0.0393 ...
# $ z: num 0.1 0.111 0.122 0.135 0.15 ...

# join with the 'metadata'
params$id <- as.character(1:nrow(params))

d <- left_join(params, all, by='id')
str(d)
# 'data.frame':	1800 obs. of 7 variables:
# $ a : num 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 ...
# $ b : num 1 1 1 1 1 1 1 1 1 1 ...
# $ c : num 0 0 0 0 0 0 0 0 0 0 ...
# $ id: chr "1" "1" "1" "1" ...
# $ x : num 0 0.101 0.202 0.303 0.404 ...
# $ y : num 0 0.0101 0.0201 0.0298 0.0393 ...
# $ z : num 0.1 0.111 0.122 0.135 0.15 ...

# optional: reshape to long format for visualisation 
m <- pivot_longer(d, cols = c('y','z'))
library(ggplot2)
ggplot(m, aes(x, value, colour=a, linetype=factor(c), group=interaction(a,c))) +
  facet_grid(name~b, scales='free_y', labeller = label_both) +
  geom_line()

```

 ![Screenshot 2022-03-02 at 15.16.25](https://global.discourse-cdn.com/julialang/original/3X/a/d/add94cdd1f4f9e47f70252f36b010ad7234c8136.png)

I find this workflow very handy and extensible, and because it’s typically for interactive analyses the raw efficiency isn’t too much of a concern (a more realistic `my_model` may be slow for each iteration, so any slight overhead of manipulating the data this way is negligible).

In Julia, I would likely use a comprehension to loop over the combinations of parameters, e.g.

```julia
all = [my_model(x, a,b,c) for a=..., b = ..., c = ...]

```

and then splat the results together, add repeated versions of the parameters a,b,c, but it’s less streamlined. Am I missing an equivalent to `purrr::pmap_df()` and `dplyr::left_join`?  
I saw some uses of Base.Cartesian, but it doesn’t feel super intuitive to me.

Many thanks.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [March 2, 2022, 2:54pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/2 "2022-03-02T14:54:16Z")

</div>

This is a complicated question. DataFrames.jl can certainly handle this kind of operation.

I think one thing that’s missing in your workflow is an easy Julia equivalent of `expand.grid`. This function should suffice

```julia
function expand_grid(; kws...)
    names, vals = keys(kws), values(kws)
    return DataFrame(NamedTuple{names}(t) for t in Iterators.product(vals...))
end

```

Your `my_model` function can be ported almost directly

```julia
function mymodel(;x = range(0, 10, length = 100), a = 1, b = 1, c = 2, fun = sin)
    DataFrame(x = x, y = a .* sin.(b .* x) .+ c, z = a .* fun.(b .* x) .+ c)
end

```

So here is stuff so far

```julia
julia> function mymodel(;x = range(0, 10, length = 100), a = 1, b = 1, c = 2, fun = sin)
           DataFrame(x = x, y = a .* sin.(b .* x) .+ c, z = a .* fun.(b .* x) .+ c)
       end
mymodel (generic function with 6 methods)

julia> params = expand_grid(a=[0.1,0.2,0.3], b = [1,2,3], c = [0,0.5]);

julia> map(eachrow(params)) do r
           mymodel(;fun = tanh, pairs(r)...)
       end;

```

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [March 2, 2022, 2:55pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/3 "2022-03-02T14:55:18Z")

</div>

If I understand you correctly then what I tend to do is:

```julia
julia> param1 = [1,2,3]; param2 = ["a", "b"];

julia> df = rename!(DataFrame(IterTools.product(param1, param2)), ["param1", "param2"])

julia> df[!, :result] .= 0.0; df
6×3 DataFrame
 Row │ param1 param2 result  
     │ Int64 String Float64 
─────┼─────────────────────────
   1 │ 1 a 0.0
   2 │ 2 a 0.0
   3 │ 3 a 0.0
   4 │ 1 b 0.0
   5 │ 2 b 0.0
   6 │ 3 b 0.0

julia> for r ∈ eachrow(df)
           r.result = mymodel(r.param1, r.param2)
       end

```

(of course `result` could be anything, i.e. it could hold a `NamedTuple` with multiple results)

---

<div class="post-metadata">

**Author:** ![baptnz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baptnz/32/24519_2.png) [@baptnz](https://discourse.julialang.org/u/baptnz)\
**Post date:** [March 2, 2022, 4:41pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/4 "2022-03-02T16:41:34Z")

</div>

Thanks! I’d used `Iterators.product` in the past (but forgotten what it was called), but I doubt I would have managed to wrap it as neatly as `expand_grid` here.

With the map over rows, what would be a good way to store the results and combine them all together with the parameters?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [March 2, 2022, 4:48pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/5 "2022-03-02T16:48:30Z")

</div>

DataFrames has `leftjoin` just like `R` so your approach in R will also work.

---

<div class="post-metadata">

**Author:** ![baptnz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baptnz/32/24519_2.png) [@baptnz](https://discourse.julialang.org/u/baptnz)\
**Post date:** [March 2, 2022, 4:48pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/6 "2022-03-02T16:48:51Z")

</div>

> [@nilshg](#):
>
> ```julia
> for r ∈ eachrow(df)
> r.result = mymodel(r.param1, r.param2)
> end
> 
> ```

Thanks, I’m curious how you’d handle the case where mymodel returns a DataFrame (with 100 rows here). Preallocating the ‘result’ column seems problematic because it’s the wrong container type and dimension. And if I can allocate it an arbitrary object, what can I use to “unnest” the DataFrame to a flat format at the end? (in R’s tidyverse, that would be:

```julia
params$results <- pmap(params, my_model, fun = tanh)
unnest(params, cols='results')
# A tibble: 1,800 × 6
       a b c x y z
   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
 1 0.1 1 0 0 0 0     
 2 0.1 1 0 0.101 0.0101 0.0101
 3 0.1 1 0 0.202 0.0201 0.0199
 4 0.1 1 0 0.303 0.0298 0.0294
 5 0.1 1 0 0.404 0.0393 0.0383
 6 0.1 1 0 0.505 0.0484 0.0466
 7 0.1 1 0 0.606 0.0570 0.0541
 8 0.1 1 0 0.707 0.0650 0.0609
 9 0.1 1 0 0.808 0.0723 0.0669
10 0.1 1 0 0.909 0.0789 0.0721
# … with 1,790 more rows

```

---

<div class="post-metadata">

**Author:** ![baptnz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baptnz/32/24519_2.png) [@baptnz](https://discourse.julialang.org/u/baptnz)\
**Post date:** [March 2, 2022, 5:02pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/7 "2022-03-02T17:02:05Z")

</div>

Then I need to have an “id” column in both `params` (easy enough), and the result of `map()` – do you see an elegant way of obtaining this?  
The brute-force way I can see is to `vcat()` the result of `map()`, and then create an id variable with the right number of repeats.

```julia

params = expand_grid(a=[0.1,0.2,0.3], b = [1,2,3], c = [0,0.5]);

all = map(eachrow(params)) do r
    mymodel(;fun = tanh, pairs(r)...)
end;

params[!, :id] = 1:nrow(params)

all_df = vcat(all...)
all_df[!, :id] = repeat(1:nrow(params), inner=100)

DataFrames.leftjoin(params, all_df, on=:id)

```

(feels a bit clunky…)

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [March 2, 2022, 5:10pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/8 "2022-03-02T17:10:05Z")

</div>

Don’t `splat` like that. In Julia, you should never splat large collections. Use `reduce(vcat, all)` instead.

You could use `mapreduce` instead of `map` to get a DataFrame out, combining both the `map` and the `reduce` into one call.

```julia
all_df[!, :id] = repeat(1:nrow(params), inner=100)

```

This is a bit scary, as it relies on the length of whatever you are using being `100`. Maybe have `id` be an input into the `mymodel` function? Just spitballing though.

---

<div class="post-metadata">

**Author:** ![baptnz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baptnz/32/24519_2.png) [@baptnz](https://discourse.julialang.org/u/baptnz)\
**Post date:** [March 2, 2022, 5:24pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/9 "2022-03-02T17:24:49Z")

</div>

> [@pdeffebach](#):
>
> Don’t `splat` like that. In Julia, you should never splat large collections. Use `reduce(vcat, all)` instead.

Thanks – I did see someone warning another that this was inefficient; reduce is basically R’s `do.call(rbind, all)`, with a better name, so I’m happy remembering that.

Edit\*: Actually, `reduce(vcat, all)` is not super intuitive for me, because I would have expected it to be super inefficient (creating recursively larger and larger intermediates until all DataFrames are combined). From `?reduce` I see that there’s a special method for vcat-like functions (?) to bypass this problem, presumably.

> [@pdeffebach](#):
>
> You could use `mapreduce` instead of `map` to get a DataFrame out, combining both the `map` and the `reduce` into one call.

Thanks – I suspected there’d be an equivalent to `map_df` (which combines the map + reduce).

> [@pdeffebach](#):
>
> ```julia
> all_df[!, :id] = repeat(1:nrow(params), inner=100)
> 
> ```
> 
> This is a bit scary, as it relies on the length of whatever you are using being `100` . Maybe have `id` be an input into the `mymodel` function? Just spitballing though.

Definitely a concern, I would not want to have that in my code. I _could_ modify `my_model` (just like I could have it include `a, b, c` in the returned DataFrame), but I’m hoping there’s a natural approach that makes it unnecessary.

---

<div class="post-metadata">

**Author:** ![baptnz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baptnz/32/24519_2.png) [@baptnz](https://discourse.julialang.org/u/baptnz)\
**Post date:** [March 3, 2022, 10:34am UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/10 "2022-03-03T10:34:15Z")

</div>

I’ve noticed that `reduce` has an optional `source` argument specifically to keep track of each block. With this, I now have the following wrapper:

```julia

function expand_grid(; kws...)
    names, vals = keys(kws), values(kws)
    return DataFrame(NamedTuple{names}(t) for t in Iterators.product(vals...))
end

function mymodel(;x = range(0, 10, length = 100), a = 1, b = 1, c = 2, fun = sin, kws...)
    DataFrame(x = x, y = a .* sin.(b .* x) .+ c, z = a .* fun.(b .* x) .+ c)
end

function pmap_df(p, f, kws...; join=true)
   tmp = map(f, eachrow(p), kws...)
   all = reduce(vcat, tmp, source="id")
   if !join
    return all
   end
   p[!, :id] = 1:nrow(p)
   return DataFrames.leftjoin(p, all, on=:id)
end

params = expand_grid(a=[0.1,0.2,0.3], b = [1,2,3], c = [0,0.5]);

pmap_df(params, p -> mymodel(;p..., fun=tanh))

```

That’s pretty close to what I was after; I’d still like to merge the map+reduce steps with `mapreduce`, but I’m not sure how to pass extra arguments to `reduce`. Any idea?

---

<div class="post-metadata">

**Author:** ![spaceLem](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/spacelem/32/217628_2.png) [@spaceLem](https://discourse.julialang.org/u/spaceLem)\
**Post date:** [April 29, 2022, 10:55pm UTC](https://discourse.julialang.org/t/map-over-combinations-of-parameters-and-grouping-results-as-dataframe/77306/11 "2022-04-29T22:55:43Z")

</div>

> [@pdeffebach](#):
>
> ```julia
> function expand_grid(; kws...)
> names, vals = keys(kws), values(kws)
> return DataFrame(NamedTuple{names}(t) for t in Iterators.product(vals...))
> end
> 
> ```

I was just searching for a Julia equivalent of R’s `expand.grid`, and that is _exactly_ what I needed, and in such a lovely concise way. Thanks!