# Newbie question about loop fusion and broadcasting

**URL:** <https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210>\
**Category:** New to Julia\
**Created:** [June 11, 2020, 4:52pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210 "2020-06-11T16:52:09Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![cwindolf](https://avatars.discourse-cdn.com/v4/letter/c/ecd19e/32.png) [@cwindolf](https://discourse.julialang.org/u/cwindolf)\
**Post date:** [June 11, 2020, 4:52pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/1 "2020-06-11T16:52:09Z")

</div>

Dear JuliaLang,

I’m excited to get started with Julia as a long-time Python user, tired of typing `np.array`. One thing that is really exciting to me is the ability to avoid writing my own `ufunc`s, especially avoiding the slow but easy to write `frompyfunc` ones.

One thing that is confusing me right now is the difference between, say, `*` and `.*`. Say I let `x = randn(10)`, and I want to compute the outer product: `x * x'`, aka `x .* x'`. These are of course the same, but from what I can tell they have different performance characteristics when combined with other broadcasted operations – the `.*` form can be fused with other broadcasted operations while `*` cannot.

However, to me, it seems like both of these are doing broadcasting, and that there is no room to interpret them differently from a mathematical point of view. So, is there an intuition for what underlying factor makes the difference for loop fusion? Is there just some loop fusion switch that gets turned on only when a `.` is present?

Thanks!

---

<div class="post-metadata">

**Author:** ![klaff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klaff/32/7637_2.png) [@klaff](https://discourse.julialang.org/u/klaff)\
**Post date:** [June 11, 2020, 5:03pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/2 "2020-06-11T17:03:36Z")

</div>

It turns out they are both doing broadcasting, but not by a trick of syntax.

If you use `@edit x * x'`, you can see that the called method of `*` explicitly calls broadcast.

From adjtrans.jl line 272:

```julia
# vector * Adjoint/Transpose-vector
*(u::AbstractVector, v::AdjOrTransAbsVec) = broadcast(*, u, v)

```

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [June 11, 2020, 5:06pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/3 "2020-06-11T17:06:51Z")

</div>

> Is there just some loop fusion switch that gets turned on only when a `.` is present?

Yes, that’s precisely it. Broadcast’s loop fusion is a _syntax_ thing. While Julia’s compiler could maybe someday get smart enough to look inside the `*` method to see that it’s “just” a broadcast — and thus can avoid allocating its temporary, it requires a rather complex escape analysis to make sure that this intermediate isn’t stored/used/re-used/changed/etc and a complex function analysis to ensure that it actually is doing something that is broadcast-compatible.

On the other hand, just looking at an expression and combining the `.`s is a very simple pass.

---

<div class="post-metadata">

**Author:** ![klaff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klaff/32/7637_2.png) [@klaff](https://discourse.julialang.org/u/klaff)\
**Post date:** [June 11, 2020, 5:07pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/4 "2020-06-11T17:07:40Z")

</div>

I should add that `a .* b` is syntax for `broadcast(*, a, b)`. See [More Dots: Syntactic Loop Fusion in Julia](https://julialang.org/blog/2017/01/moredots/) for a good explanation.

---

<div class="post-metadata">

**Author:** ![cwindolf](https://avatars.discourse-cdn.com/v4/letter/c/ecd19e/32.png) [@cwindolf](https://discourse.julialang.org/u/cwindolf)\
**Post date:** [June 11, 2020, 5:55pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/5 "2020-06-11T17:55:39Z")

</div>

Thanks all for the interesting responses! I can tell this is a nice community. The “More Dots” article was especially helpful.

To make sure I’m getting it: that article says that dot-calls are sugar for `broadcast`. But, if I’m understanding right, it also seems to indicate that dots are explicitly required in order for loop fusion to kick in. So, although (as @klaff writes) `x .* x'` and `x * x'` both end up mapping onto calls to `broadcast`, only the former will trigger loop fusion (if there were more vectorized operations involved). Is that the right idea?

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [June 11, 2020, 6:01pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/6 "2020-06-11T18:01:26Z")

</div>

> [@cwindolf](#):
>
> Is that the right idea?

Yup!

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [June 11, 2020, 6:07pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/7 "2020-06-11T18:07:56Z")

</div>

Yes, that’s exactly right. For example: `v .* v' .+ 2` is effectively the same as:

```julia
vt = v'
broadcast((x, xt)->x*xt + 2, v, vt)

```

Had I written this as `v * v' .+ 2`, then we’d first compute the result of `*` (regardless of the implementation of `*`) and then do the broadcasted addition, effectively:

```julia
temp = v * v'
broadcast(x->x+2, temp)

```

The consecutively-connected dots are how we define what goes into that inner “kernel” (the anonymous function in the cartoon equivalence above). To make this even more clear, you can compare the result of `zeros(5) .+ rand()` vs. `zeros(5) .+ rand.()`.

---

<div class="post-metadata">

**Author:** ![cwindolf](https://avatars.discourse-cdn.com/v4/letter/c/ecd19e/32.png) [@cwindolf](https://discourse.julialang.org/u/cwindolf)\
**Post date:** [June 11, 2020, 6:10pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/8 "2020-06-11T18:10:47Z")

</div>

> [@mbauman](#):
>
> `zeros(5) .+ rand.()`

Whoah… no way…

---

<div class="post-metadata">

**Author:** ![klaff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klaff/32/7637_2.png) [@klaff](https://discourse.julialang.org/u/klaff)\
**Post date:** [June 11, 2020, 7:23pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/9 "2020-06-11T19:23:28Z")

</div>

> [@cwindolf](#):
>
> Thanks all for the interesting responses! I can tell this is a nice community.

You’re welcome! But also, you asked a very astute question. I thought I understood dot fusion but didn’t really understand the purely syntactical nature of it.

---

<div class="post-metadata">

**Author:** ![klaff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klaff/32/7637_2.png) [@klaff](https://discourse.julialang.org/u/klaff)\
**Post date:** [June 12, 2020, 3:47pm UTC](https://discourse.julialang.org/t/newbie-question-about-loop-fusion-and-broadcasting/41210/10 "2020-06-12T15:47:16Z")

</div>

This is still making my brain hurt so here’s a question:  
I think  
`zeros(5) .+ rand()` is equivalent to  
`(+).(zeros(5), rand())` is equivalent to  
`broadcast(+, zeros(5), rand())`.

Is this the right way to think about this one?  
`zeros(5) .+ rand.()` is equivalent to  
`(+).(zeros(5),rand.())` is equivalent to  
`(x -> x+rand()).(zeros(5))` is equivalent to [Edit - added this line]  
`broadcast(x -> x+rand(), zeros(5))`?