# Multi-threading with DataFrames

**URL:** <https://discourse.julialang.org/t/multi-threading-with-dataframes/101859>\
**Category:** General Usage\
**Tags:** multithreading, dataframes\
**Created:** [July 20, 2023, 10:16pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859 "2023-07-20T22:16:35Z")\
**Posts on this page:** 14\
**Page:** 2

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [December 21, 2023, 7:53am UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/21 "2023-12-21T07:53:01Z")

</div>

CC @pdeffebach

But I think the only thing you can do is to add `@transform` to the expression and then evaluate it

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 21, 2023, 12:21pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/22 "2023-12-21T12:21:53Z")

</div>

You can parse the `:c = 2 * :a + :b` to become a `src => fun => dest` pair that you can pass around and use later. Use the (currently un-exported) `DataFramesMeta.@col` feature

```julia
julia> using DataFramesMeta

julia> df = DataFrame(a = 1, b = 2);

julia> t = DataFramesMeta.@col :c = 2 * :a + :b
[:a, :b] => (var"#3#4"() => :c)

julia> transform(df, t)
1×3 DataFrame
 Row │ a b c
     │ Int64 Int64 Int64
─────┼─────────────────────
   1 │ 1 2 4

```

I’ve been meaning to export this feature, since it is useful.

---

<div class="post-metadata">

**Author:** ![sob](https://avatars.discourse-cdn.com/v4/letter/s/65b543/32.png) [@sob](https://discourse.julialang.org/u/sob)\
**Post date:** [December 21, 2023, 7:12pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/23 "2023-12-21T19:12:14Z")

</div>

Thing is in this case the transformation I have as an expression in a variable, which I can’t pass as an argument to `@col`

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 21, 2023, 7:13pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/24 "2023-12-21T19:13:04Z")

</div>

If that’s the case I think you should re-think your whole approach. Use functions to store transformations, not expressions. You should not be passing around expressions like that.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [December 21, 2023, 8:44pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/25 "2023-12-21T20:44:59Z")

</div>

perhaps if you provided more details about the problem you face, more suggestions might come.  
Could the [symbolics.jl](https://symbolics.juliasymbolics.org/stable/getting_started/) package be useful to you in your case?

---

<div class="post-metadata">

**Author:** ![sob](https://avatars.discourse-cdn.com/v4/letter/s/65b543/32.png) [@sob](https://discourse.julialang.org/u/sob)\
**Post date:** [December 21, 2023, 10:23pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/26 "2023-12-21T22:23:22Z")

</div>

Certainly, I’m not sure it will be of much help (compared to what I mentioned in above posts) given the generic nature of what I’m trying to achieve, but let me do it anyway.

I have `DataFrames` of time series data, over which the end user of the program (very often, me) performs high numbers of transformations for further analysis. These transformations do vary and are best stored as lists of Julia `Expr` (made from strings). I realize that this is essentially me using a subset of the Julia language as my own language that is exposed to the end user, hence no way around `eval()` unless I describe my transformations ahead of time in very generic terms and reduce flexibility.

So with the example `DataFrame` above, I want the user to be able to apply say the 4 operations, sine/cosine, exp, log, and more (some being defined in my module), with any arbitraty combination possible. I’m essentially recreating a small calculator that is applied column-wise to a `DataFrame`. Hope this makes sense. Appreciate you reading my convoluted problem.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 21, 2023, 10:32pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/27 "2023-12-21T22:32:42Z")

</div>

This is absolutely feasible with functions. And you should definitely be using them instead of expressions.

```julia
julia> df = DataFrame(a = [1, 2, 3]; b = [4, 5, 6]);

julia> function operate_on_col(fun, col1, col2)
           DataFramesMeta.@col @byrow :newcol = begin
               $col1 * 2 + fun($col2)
           end
       end;

julia> t1 = operate_on_col(sin, :a, :b); t2 = operate_on_col(cos, :a, :b);

julia> transform(df, t1)
3×3 DataFrame
 Row │ a b newcol  
     │ Int64 Int64 Float64 
─────┼───────────────────────
   1 │ 1 4 1.2432
   2 │ 2 5 3.04108
   3 │ 3 6 5.72058

julia> transform(df, t2)
3×3 DataFrame
 Row │ a b newcol  
     │ Int64 Int64 Float64 
─────┼───────────────────────
   1 │ 1 4 1.34636
   2 │ 2 5 4.28366
   3 │ 3 6 6.96017

```

---

<div class="post-metadata">

**Author:** ![sob](https://avatars.discourse-cdn.com/v4/letter/s/65b543/32.png) [@sob](https://discourse.julialang.org/u/sob)\
**Post date:** [December 21, 2023, 10:34pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/28 "2023-12-21T22:34:22Z")

</div>

Thank you for the suggestion, I think my issue is `fun` is not known ahead of time, but let me take a closer look at your code see if I can leverage that. (And by “not known” I mean “not defined”).

In your example, I would pretty much need to have the user pass the definition of `fun` as a string and `eval()` its definition, which takes me back to square one.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 21, 2023, 10:51pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/29 "2023-12-21T22:51:40Z")

</div>

Could you not have the user pass `fun`? They can’t make it themselves?

---

<div class="post-metadata">

**Author:** ![sob](https://avatars.discourse-cdn.com/v4/letter/s/65b543/32.png) [@sob](https://discourse.julialang.org/u/sob)\
**Post date:** [December 21, 2023, 11:04pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/30 "2023-12-21T23:04:58Z")

</div>

The user lives outside of Julia, meaning they can pass as string, which would have to be parsed and eval’d?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [December 22, 2023, 1:25am UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/31 "2023-12-22T01:25:17Z")

</div>

Yeah that would most likely have to be parsed and eval’d, which isn’t ideal. Maybe someone else can chime in on the best way to do that particular task.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [December 22, 2023, 11:18pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/32 "2023-12-22T23:18:56Z")

</div>

```julia
julia> using DataFrames, Symbolics

julia> df_flows = DataFrame(;
           from = ["p1", "p1", "p2", "p2", "p1", "p2"],
           to = ["d", "d", "d", "d", "d", "d"],
           rp = [1, 1, 1, 1, 2, 2],
           tb = [3,5,7,4,6,8],
           index = 1:6,
       )
6×5 DataFrame
 Row │ from to rp tb index 
     │ String String Int64 Int64 Int64 
─────┼─────────────────────────────────────
   1 │ p1 d 1 3 1
   2 │ p1 d 1 5 2
   3 │ p2 d 1 7 3
   4 │ p2 d 1 4 4
   5 │ p1 d 2 6 5
   6 │ p2 d 2 8 6

julia> @variables x y z
3-element Vector{Num}:
 x
 y
 z

julia> w=x^2+y*sqrt(z*x)
x^2 + y*sqrt(x*z)

julia> transform(df_flows,[3,4,5]=>ByRow((r,t,i)->substitute(w, Dict(x=>r,y=>t,z=>i))))
6×6 DataFrame
 Row │ from to rp tb index rp_tb_index_function   
     │ String String Int64 Int64 Int64 Num
─────┼───────────────────────────────────────────────────────────  
   1 │ p1 d 1 3 1 4.0       
   2 │ p1 d 1 5 2 8.07107   
   3 │ p2 d 1 7 3 13.1244    
   4 │ p2 d 1 4 4 9.0       
   5 │ p1 d 2 6 5 22.9737    
   6 │ p2 d 2 8 6 31.7128    

julia> w=3x-2y
3x - 2y

julia> transform(df_flows,[3,4,5]=>ByRow((r,t,i)->substitute(w, Dict(x=>r,y=>t,z=>i))))
6×6 DataFrame
 Row │ from to rp tb index rp_tb_index_function   
     │ String String Int64 Int64 Int64 Num
─────┼───────────────────────────────────────────────────────────  
   1 │ p1 d 1 3 1 -3   
   2 │ p1 d 1 5 2 -7   
   3 │ p2 d 1 7 3 -11   
   4 │ p2 d 1 4 4 -5   
   5 │ p1 d 2 6 5 -6   
   6 │ p2 d 2 8 6 -10   

```

PS  
I saw several [old posts](https://discourse.julialang.org/t/convert-string-to-function-name/1547) that discussed the problem of deriving a function from a string. Apart from the solution with parse() and eval() I have seen the use of the getfield function applied to the current module to obtain the function from the string.  
But this applies to functions already defined in the module.  
An equivalent would be, in my opinion, a dict with strings as keys and functions as values.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [December 23, 2023, 8:47am UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/33 "2023-12-23T08:47:04Z")

</div>

to get suggestions for the current problem (about the use of an input string) it might be more useful to open a new topic with a more specific title

In principle a “simple calculator” seems feasible.  
There’s a lot of work to do to make it truly functional, but just to start a seed

```julia

df=Dict("log"=>log, "sin"=>sin, "*"=>*,"+"=>+,"-"=>-,"^"=>^, "∘"=>∘)

function str2func(str)
    lff=findfirst('(', str)
    op=df[str[1:lff-1]]
    if !occursin('(',str[lff+1:end])
        par=split(str[lff+1:end-1],',')
        tp=tryparse.(Int,par)
        if all(isnothing,tp)
            return ((x,y)->(a->(b->(c->op(a,c))(b)))(x)(y))
        else
            n=only(filter(!isnothing,tp))
            return z->((x,y)->(a->(b->(c->op(a,c))(b)))(x)(y))(n,z)
        end
    else
        return (x...)->"not yet"
    end
end

julia> df_flows
6×5 DataFrame
 Row │ from to rp tb index 
     │ String String Int64 Int64 Int64
─────┼─────────────────────────────────────
   1 │ p1 d 1 3 1
   2 │ p1 d 1 5 2
   3 │ p2 d 1 7 3
   4 │ p2 d 1 4 4
   5 │ p1 d 2 6 5
   6 │ p2 d 2 8 6

julia> transform(df_flows,[3,4]=>ByRow(str2func("*(x,y)")))    
6×6 DataFrame
 Row │ from to rp tb index rp_tb_function     
     │ String String Int64 Int64 Int64 Int64
─────┼─────────────────────────────────────────────────────    
   1 │ p1 d 1 3 1 3     
   2 │ p1 d 1 5 2 5     
   3 │ p2 d 1 7 3 7     
   4 │ p2 d 1 4 4 4     
   5 │ p1 d 2 6 5 12     
   6 │ p2 d 2 8 6 16     

julia> transform(df_flows,[3,4]=>ByRow(str2func("+(x,y)")))    
6×6 DataFrame
 Row │ from to rp tb index rp_tb_function     
     │ String String Int64 Int64 Int64 Int64
─────┼─────────────────────────────────────────────────────    
   1 │ p1 d 1 3 1 4     
   2 │ p1 d 1 5 2 6     
   3 │ p2 d 1 7 3 8     
   4 │ p2 d 1 4 4 5     
   5 │ p1 d 2 6 5 8     
   6 │ p2 d 2 8 6 10     

julia> transform(df_flows,[4,3]=>ByRow(str2func("^(x,y)")))    
6×6 DataFrame
 Row │ from to rp tb index tb_rp_function     
     │ String String Int64 Int64 Int64 Int64
─────┼─────────────────────────────────────────────────────    
   1 │ p1 d 1 3 1 3     
   2 │ p1 d 1 5 2 5     
   3 │ p2 d 1 7 3 7     
   4 │ p2 d 1 4 4 4     
   5 │ p1 d 2 6 5 36     
   6 │ p2 d 2 8 6 64     

julia> transform(df_flows,[4,3]=>ByRow(str2func("-(x,y)")))    
6×6 DataFrame
 Row │ from to rp tb index tb_rp_function     
     │ String String Int64 Int64 Int64 Int64
─────┼─────────────────────────────────────────────────────    
   1 │ p1 d 1 3 1 2     
   2 │ p1 d 1 5 2 4     
   3 │ p2 d 1 7 3 6     
   4 │ p2 d 1 4 4 3     
   5 │ p1 d 2 6 5 4     
   6 │ p2 d 2 8 6 6     

julia> transform(df_flows,[4]=>ByRow(str2func("*(3,y)")))      
6×6 DataFrame
 Row │ from to rp tb index tb_function        
     │ String String Int64 Int64 Int64 Int64
─────┼──────────────────────────────────────────────────       
   1 │ p1 d 1 3 1 9        
   2 │ p1 d 1 5 2 15        
   3 │ p2 d 1 7 3 21        
   4 │ p2 d 1 4 4 12        
   5 │ p1 d 2 6 5 18        
   6 │ p2 d 2 8 6 24        

julia> transform(df_flows,[4,3]=>ByRow(str2func("+(log(x),∘(sin, *(10,y)))")))
6×6 DataFrame
 Row │ from to rp tb index tb_rp_function     
     │ String String Int64 Int64 Int64 String
─────┼─────────────────────────────────────────────────────    
   1 │ p1 d 1 3 1 not yet
   2 │ p1 d 1 5 2 not yet
   3 │ p2 d 1 7 3 not yet
   4 │ p2 d 1 4 4 not yet
   5 │ p1 d 2 6 5 not yet
   6 │ p2 d 2 8 6 not yet

```

the functions thus defined can be, in appropriate cases (associative operators), applied to more than 2 elements.  
And in particular in the case of operators that have methods also defined on vectors, you can do without wrapping everything with ByRow

```julia
transform(df_flows,[3,4, 5]=>ByRow((x...)->reduce(str2func("+(x,y)"),x)))

transform(df_flows,[3,4, 5]=>(x...)->foldl(str2func("+(x,y)"),x))

```

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [December 24, 2023, 3:36pm UTC](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859/34 "2023-12-24T15:36:02Z")

</div>

a small step forward(?). If you illustrate the typical cases of formulas used, perhaps something can be added

I realize that the idea and above all the implementation is really naive and, I fear, not very efficient.  
But let’s play with Julia’s expressions and while waiting for ideas for improvement, I’ll give you a small step forward(?).

If you illustrate the typical cases of formulas used, perhaps something can be added

```julia

julia> function str2func(str)
           lff=findfirst('(', str)
           op=df[str[1:lff-1]]
           if !occursin('(',str[lff+1:end])
               par=split(str[lff+1:end-1],',')
               tp=tryparse.(Int,par)
               opxy=(x,y)->(b->(c->op(c,b)))(x)(y)
               if all(isnothing,tp)
                   return opxy
               else
                   n=only(filter(!isnothing,tp))
                   return z->opxy(n,z)
               end
           else
               par=split(str[lff+1:end-1],"),")
               par[1:end-1] .*=')'
               return (x...)->op([str2func(p)(var) for (p,var) in zip(par,x)]...)
           end
       end
str2func (generic function with 1 method)

julia> str= "+(*(3,x),*(2,y))"
"+(*(3,x),*(2,y))"

julia> str2func(str)(3,3)
15

julia> str= "*(+(3,x),+(2,y))"
"*(+(3,x),+(2,y))"

julia> str2func(str)(2,3)
25

julia> str= "+(3,x)"
"+(3,x)"

julia> str2func(str)(-3)
0

```

I posted an updated version of the script [here](https://discourse.julialang.org/t/convert-string-to-function-name/1547/30)

[Previous page](https://discourse.julialang.org/t/multi-threading-with-dataframes/101859.md?page=1)