# Concatenate DataFrame columns dynamically

**URL:** <https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106>\
**Category:** General Usage\
**Tags:** dataframes\
**Created:** [September 24, 2019, 12:37pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106 "2019-09-24T12:37:00Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [September 24, 2019, 12:37pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/1 "2019-09-24T12:37:00Z")

</div>

Hello, amazing Julia community! 😃

I’m stumped this morning on what I thought would be a fairly trivial task: I am trying to write a function that creates a new column in a DataFrame, the values of which are simply the concatenation of some other columns in the DataFrame. If I know the columns upfront, it is indeed a very trivial task. However, I want this function to take a DataFrame as one argument and then the column numbers to concatenate as another…something like this:

```julia
function add_column(df::DataFrame, colname::Symbol, cols_to_concat::Vector{Integer})
    df[!,colname] = df[!, 1] .* df[!, 2] .* df[!, 3]
end

# OR 

function add_column(df::DataFrame, colname::Symbol, cols_to_concat::Vector{Integer})
    df[!,colname] = ["$(row[cols_to_concat[1]])$(row[cols_to_concat[2]])" for row in eachrow(df)]
end

```

I’m thinking there may be a metaprogramming solution to this? It seems to me that the issue is figuring out how to dynamically construct this piece `df[!, 1] .* df[!, 2] .* df[!, 3]` or the alternative version in the second function. I think I can dynamically construct the `df[!, n]` pieces fairly easily, like this:

```julia
dfcols = [Symbol("df[!, $n]") for n in cols_to_concat]

```

but I’m not sure then how to piece them together with the `.*` in between…

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [September 24, 2019, 12:51pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/2 "2019-09-24T12:51:42Z")

</div>

It looks like I can do this:

```julia
cols_to_concat = [1,3,5]
dfcols = [i == length(cols_to_concat) ? Symbol("df[!, $n]") : Symbol("df[!, $n] .* ") for (i, n) in enumerate(cols_to_concat)]

```

but I am still unable to figure out the last piece. I tried:

```julia
@eval dfcols...

```

but that doesn’t work. If I simply print it out, I get the desired output:

```julia
julia> println(dfcols...)
df[!, 1] .* df[!, 3] .* df[!, 5]

```

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [September 24, 2019, 1:09pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/3 "2019-09-24T13:09:23Z")

</div>

I’m probably misunderstanding the question, but does this help:

```julia
using DataFrames

df = DataFrame(a = ["aa" for _ in 1:10], b = ["bb" for _ in 1:10], 
               c = ["cc" for _ in 1:10], d = ["dd" for _ in 1:10])

function add_column(df::DataFrame, colname::Symbol, cols_to_concat::Vector)
   df[!,colname] = (df[!, names(df)[cols_to_concat[1]]] 
                .* df[!, names(df)[cols_to_concat[2]]] 
                .* df[!, names(df)[cols_to_concat[3]]])
end

add_column(df, :add, [1, 4, 4])

```

Output:

```julia
10×5 DataFrame
│ Row │ a │ b │ c │ d │ add │
│ │ String │ String │ String │ String │ String │
├─────┼────────┼────────┼────────┼────────┼────────┤
│ 1 │ aa │ bb │ cc │ dd │ aadddd │
│ 2 │ aa │ bb │ cc │ dd │ aadddd │
│ 3 │ aa │ bb │ cc │ dd │ aadddd │
│ 4 │ aa │ bb │ cc │ dd │ aadddd │
│ 5 │ aa │ bb │ cc │ dd │ aadddd │
│ 6 │ aa │ bb │ cc │ dd │ aadddd │
│ 7 │ aa │ bb │ cc │ dd │ aadddd │
│ 8 │ aa │ bb │ cc │ dd │ aadddd │
│ 9 │ aa │ bb │ cc │ dd │ aadddd │
│ 10 │ aa │ bb │ cc │ dd │ aadddd │

```

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [September 24, 2019, 1:15pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/4 "2019-09-24T13:15:28Z")

</div>

Hi! I need this to work though when there are more or less than 3 columns. Your function will concatenate the first three columns that are given in the `cols_to_concat` vector, but I need to write the function so that the `cols_to_concat` vector could contain 1, 3, 12, etc., columns that need to be concatenated…

I tried this, but it doesn’t work:

```julia
function add_column(df::DataFrame, colname::Symbol, cols_to_concat::Vector{Int64})
    dfcols = [i == length(cols_to_concat) ? Symbol("df[!, $n]") : Symbol("df[!, $n] .* ") for (i, n) in enumerate(cols_to_concat)]
    left_side = Symbol("$(df).$(colname)")
    right_side = Symbol(string(dfcols...))
    @eval $left_side = $right_side
    return df
end

julia> add_column(df1, :new, [1,3,5,6])
ERROR: UndefVarError: df[!, 1] .* df[!, 3] .* df[!, 5] .* df[!, 6] not defined
Stacktrace:
 [1] top-level scope at REPL[20]:1
 [2] eval at .\boot.jl:330 [inlined]
 [3] add_column(::DataFrame, ::Symbol, ::Array{Int64,1}) at .\REPL[18]:5
 [4] top-level scope at REPL[20]:1

```

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [September 24, 2019, 1:38pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/5 "2019-09-24T13:38:47Z")

</div>

Not perfect, but you will get the idea:

```julia
function add_column(df::DataFrame, colname::Symbol, cols_to_concat::Vector)
	exstring = "df[!, :" * string(colname) *"] = ("
	exstring *= "df[!, names(df)["*string(cols_to_concat[1])*"]]"
	for col in 2:length(cols_to_concat)
		exstring *= ".* df[!, names(df)["*string(cols_to_concat[col])*"]]"
	end
	exstring *= ")"
	eval(Meta.parse(exstring))
end

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 24, 2019, 1:48pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/6 "2019-09-24T13:48:39Z")

</div>

using `eval` and `parse` is way overthinking this.

```julia
function add_column!(df::DataFrame, colname::Symbol, cols_to_concat::Vector)
	t = df[!, cols_to_concat[1]]
	for col in cols_to_concat[2:end]
		t = t .* df[!, col]
	end
	df[!, colname] = t
	return df
end

```

You could also use `reduce` for this.

```julia
function add_column2!(df::DataFrame, colname::Symbol, cols_to_concat::Vector)
	function concat(x, y)
		x .* y
	end
	df[!, colname] =reduce(concat, eachcol(df[!, cols_to_concat]))
	return df
end

```

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [September 24, 2019, 2:06pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/7 "2019-09-24T14:06:31Z")

</div>

> using `eval` and `parse` is way overthinking this.

But it’s so cool!!! 😝

Seriously though, thanks so much for this brilliantly simple solution. Let me see if I can walk through how it actually works:

```julia
function add_column!(df::DataFrame, colname::Symbol, cols_to_concat::Vector)
	t = df[!, cols_to_concat[1]] #1
	for col in cols_to_concat[2:end] #2
		t = t .* df[!, col] #3
	end #4
	df[!, colname] = t #5
	return df #6
end

```

On line 1, we are grabbing column number `cols_to_concat[1]` (whatever it may be) and assigning it to a variable `t`. Then, on lines 2 - 4, we are looping through the rest of the columns in `cols_to_concat` and updating the value of `t` on each iteration of the loop by concatenating an additional column until we arrive at the end. Finally, on lines 5 and 6 we create the new column in `df`, assign it the final value of `t` and return `df`…why didn’t I think of that?! 🙃

Basically, I was trying to figure out how to do it all at once when there was no need - your way simply concatenates the columns in an iterative fashion 👍👍

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 24, 2019, 2:10pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/8 "2019-09-24T14:10:26Z")

</div>

I’m glad I could help. You have the right interpretation.

I _think_ it might be the case that the second solution I posted is faster. But the compiler might be smart and treat the two functions as equal.

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [September 24, 2019, 2:16pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/9 "2019-09-24T14:16:30Z")

</div>

Awesome, thanks again!

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [September 24, 2019, 5:49pm UTC](https://discourse.julialang.org/t/concatenate-dataframe-columns-dynamically/29106/10 "2019-09-24T17:49:14Z")

</div>

Just a general note here because I think this is a good example for a broader point. If you strip away everything about the text being stored in a DataFrame, dynamic column selection, etc… and just imagine that each element was stored in the same vector of strings (a much simpler problem), you would likely arrive immediately at the solution. This is what is nice about Julia, no need to get fancy, even when the problem looks fancy.

```julia
text = ["aa"; "bb"; "cc"]
function replconcat(str)
    final = str[1]
    for i=2:length(str)
        final = final .* str[i]
    end
    final
end
replconcat(text)

```

So to apply it to your problem just map appropriately, which is to think about doing this for each row of a DataFrame, which is made easier by the fact that the operation does not necessarily have to be done with a loop as @pdeffebach showed.