# Creating a macro to broadcast in parallel

**URL:** <https://discourse.julialang.org/t/creating-a-macro-to-broadcast-in-parallel/53508>\
**Category:** Performance\
**Tags:** macros, parallel, dataframes\
**Created:** [January 17, 2021, 11:58pm UTC](https://discourse.julialang.org/t/creating-a-macro-to-broadcast-in-parallel/53508 "2021-01-17T23:58:40Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![s-kap](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@s-kap](https://discourse.julialang.org/u/s-kap)\
**Post date:** [January 17, 2021, 11:58pm UTC](https://discourse.julialang.org/t/creating-a-macro-to-broadcast-in-parallel/53508/1 "2021-01-17T23:58:40Z")

</div>

Hey all,

I’m trying to create a macro to broadcast in parallel. I have a large dataframe and I frequently create new columns based on values from old columns.

Let’s take string concatenation for example.  
The non parallel broadcasting version looks like this:

```julia
df.input .* df.target

```

and the parallel broadcasted version looks like this

```julia
function divide_range_into_max_n_chunks(r::UnitRange{Int64}, n::Int64)
    range_length = r.stop - r.start
    chunk_size = Int64(ceil(range_length / n))
    return [i:min(i + chunk_size - 1, r.stop) for i in r.start:chunk_size:r.stop]
end

df_length = nrow(df)
concatenated_col = Vector(undef, df_length)

@threads for subrange in divide_range_into_max_n_chunks(1:df_length, nthreads())
    concatenated_col[subrange] = view(df.input, subrange) .* view(df.target, subrange)
end

```

I would like to turn the code up there into a macro like this

```julia
@parallel_broadcast df.input .* df.target

```

but I’m very new to this and I’m not sure what I’m doing wrong. I piped the code above into Meta.show\_sexpr and came up with this, but it doesn’t quite do the job. Here is what I have so far:

```julia
macro parallel_broadcast(col_a, broadcasted_operator, col_b)
    return (:macrocall, Symbol("@threads"), :(#= none:1 =#), (:for, (:(=), :subrange, (:call, :divide_range_into_max_n_chunks, (:call, :(:), 1, :df_length), (:call, :nthreads))), (:block,
      :(#= none:2 =#),
      (:(=), (:ref, :result_col, :subrange), (:call, broadcasted_operator, (:call, :view, col_a, :subrange), (:call, :view, col_b, :subrange)))
    )))
end

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [January 18, 2021, 12:22am UTC](https://discourse.julialang.org/t/creating-a-macro-to-broadcast-in-parallel/53508/2 "2021-01-18T00:22:24Z")

</div>

I cannot help, but I am curious: is there any reason to use a macro instead of wrapping that code into a function?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [January 18, 2021, 12:55am UTC](https://discourse.julialang.org/t/creating-a-macro-to-broadcast-in-parallel/53508/3 "2021-01-18T00:55:21Z")

</div>

How tall your dataframe has to be for this to pay back?

If you have a function (that does a lot of computation) I can understand but if your macro is just targeting `+ - * /` etc I doubt it will be worthwhile.

---

<div class="post-metadata">

**Author:** ![Satvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/satvik/32/20486_2.png) [@Satvik](https://discourse.julialang.org/u/Satvik)\
**Post date:** [January 18, 2021, 3:27am UTC](https://discourse.julialang.org/t/creating-a-macro-to-broadcast-in-parallel/53508/4 "2021-01-18T03:27:47Z")

</div>

For anything that can be done using `map` instead of `broadcast`, you can use the `ThreadsX` library:

```julia
using ThreadsX
ThreadsX.map(*, df.input, df.target)

```

---

<div class="post-metadata">

**Author:** ![N5N3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/n5n3/32/17663_2.png) [@N5N3](https://discourse.julialang.org/u/N5N3)\
**Post date:** [January 18, 2021, 6:50am UTC](https://discourse.julialang.org/t/creating-a-macro-to-broadcast-in-parallel/53508/5 "2021-01-18T06:50:20Z")

</div>

I made a package [ExBroadcast.jl](https://github.com/N5N3/ExBroadcast.jl) for similar purpose(unreleased, just for self usage). And any broadcast could be parallized by adding a `@mtb`.

```julia
using ExBroadcast
@mtb df.input .* df.target # dot expression is supported
@mtb broadcast(*, df.input, df.target) # also the call version

```

I hope this help you.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [January 18, 2021, 9:57am UTC](https://discourse.julialang.org/t/creating-a-macro-to-broadcast-in-parallel/53508/6 "2021-01-18T09:57:06Z")

</div>

You may be interested in Strided.jl which does pretty much what you are proposing.  
[https://github.com/Jutho/Strided.jl](https://github.com/Jutho/Strided.jl)