# Strange performance problem

**URL:** <https://discourse.julialang.org/t/strange-performance-problem/115227>\
**Category:** Performance\
**Tags:** question, type-stability\
**Created:** [June 5, 2024, 12:05pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227 "2024-06-05T12:05:42Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![0samuraiE](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/0samuraie/32/209825_2.png) [@0samuraiE](https://discourse.julialang.org/u/0samuraiE)\
**Post date:** [June 5, 2024, 12:05pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/1 "2024-06-05T12:05:42Z")

</div>

Hi all, I am new to this forum. Nice to meet you.  
Today I encouted a strange performance problem.  
I benchmarked below codes with Benchmarktools.

```julia
NG = 2
N = NX = NY = NZ = 256
U = rand!(zeros(N + 2NG, N + 2NG, N + 2NG));
V = rand!(zeros(N + 2NG, N + 2NG, N + 2NG));
W = rand!(zeros(N + 2NG, N + 2NG, N + 2NG));
Jx = zeros(N + 2NG, N + 2NG, N + 2NG);
Jy = zeros(N + 2NG, N + 2NG, N + 2NG);
Jz = zeros(N + 2NG, N + 2NG, N + 2NG);

function plain!(U, V, W, Jx, Jy, Jz, NX, NY, NZ, NG)
    for k in NG:NZ+1+NG, j in NG:NY+1+NG, i in NG:NX+1+NG
        upp = -(U[i-1, j, k] + U[i, j, k]) / 2
        vpp = -(V[i, j, k] + V[i+1, j, k]) / 2
        wpp = -(W[i, j, k] + W[i+1, j, k]) / 2

        ux1 = (U[i-1, j, k] + U[i, j, k]) / 2
        uy1 = (U[i, j, k] + U[i, j+1, k]) / 2
        uz1 = (U[i, j, k] + U[i, j, k+1]) / 2

        Jx[i, j, k] = upp * ux1
        Jy[i, j, k] = vpp * uy1
        Jz[i, j, k] = wpp * uz1
    end
end

@benchmark plain!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG)

function no_tmp!(U, V, W, Jx, Jy, Jz, NX, NY, NZ, NG)
    for k in NG:NZ+1+NG, j in NG:NY+1+NG, i in NG:NX+1+NG
        Jx[i, j, k] = -(U[i-1, j, k] + U[i, j, k]) / 2 * (U[i-1, j, k] + U[i, j, k]) / 2
        Jy[i, j, k] = -(V[i, j, k] + V[i+1, j, k]) / 2 * (U[i, j, k] + U[i, j+1, k]) / 2
        Jz[i, j, k] = -(W[i, j, k] + W[i+1, j, k]) / 2 * (U[i, j, k] + U[i, j, k+1]) / 2
    end
end

@benchmark no_tmp!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG)

```

As you can see the difference between two functions is that tmp vairables is assigned or not.  
BenchmarkTool said to me the former is 3x faster than the latter.  
Why?

Julia Version 1.10.3  
Commit 0b4590a5507 (2024-04-30 10:59 UTC)  
Build Info:  
Official [https://julialang.org/](https://julialang.org/) release  
Platform Info:  
OS: Linux (x86\_64-linux-gnu)  
CPU: 24 × 13th Gen Intel(R) Core™ i7-13700K  
WORD\_SIZE: 64  
LIBM: libopenlibm  
LLVM: libLLVM-15.0.7 (ORCJIT, goldmont)  
Threads: 24 default, 0 interactive, 12 GC (on 24 virtual cores)  
Environment:  
JULIA\_EDITOR = code

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 5, 2024, 12:07pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/2 "2024-06-05T12:07:51Z")

</div>

> [@0samuraiE](#):
>
> BenchmarkTool said to me the former is 3x faster than the latter.

Can you give sample data that you used, so that other people can run this?

---

<div class="post-metadata">

**Author:** ![0samuraiE](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/0samuraie/32/209825_2.png) [@0samuraiE](https://discourse.julialang.org/u/0samuraiE)\
**Post date:** [June 5, 2024, 12:09pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/3 "2024-06-05T12:09:07Z")

</div>

Thank you for replying. I added sample data.

---

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [June 5, 2024, 12:12pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/4 "2024-06-05T12:12:21Z")

</div>

I tried to run your code and I got the same performance more or less on 1.10.4, can you post the output of `versioninfo()`?

---

<div class="post-metadata">

**Author:** ![0samuraiE](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/0samuraie/32/209825_2.png) [@0samuraiE](https://discourse.julialang.org/u/0samuraiE)\
**Post date:** [June 5, 2024, 12:16pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/5 "2024-06-05T12:16:14Z")

</div>

Thank you for replying. I added version info. I run both Jupyter and Repl. But same ploblem occured.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 5, 2024, 12:27pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/6 "2024-06-05T12:27:18Z")

</div>

> [@0samuraiE](#):
>
> BenchmarkTool said to me the former is 3x faster than the latter.

I get a difference of about 36%, not 3x:

```julia
julia> @btime plain!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG);
  112.766 ms (0 allocations: 0 bytes)

julia> @btime no_tmp!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG);
  153.984 ms (0 allocations: 0 bytes)

```

> [@0samuraiE](#):
>
> As you can see the difference between two functions is that tmp vairables is assigned or not.

In the first version, you perform all of your “reads” of the array _before_ you perform any “writes”. In the second version the reads and writes are interleaved. This will make a difference because the compiler doesn’t know that the `Jx, Jy, Jz` arrays are distinct from `U, V, W`.

For example, you read `U[i, j, k]` when computing `upp`, and again when computing `ux1`, `uy1`, and `uz1`. In `plain!`, the compiler probably factors out this common subexpression — it loads `U[i,j,k]` only once. However, in `no_tmp!`, because the reads and writes are interleaved, the compiler will have to load `U[i,j,k]` three times, since it may have been overwritten by the writes to the `J` arrays (which might be the same as `U` for all the compiler knows).

Interestingly, most of the difference goes away if I simply put `@inbounds` in front of the `for` loops (i.e. write `@inbounds for`).

Note also that the order of floating-point operations is different in the two cases, because:

```julia
-(U[i-1, j, k] + U[i, j, k]) / 2 * (U[i-1, j, k] + U[i, j, k]) / 2

```

is not the same thing as

```julia
upp * ux1 == (-(U[i-1, j, k] + U[i, j, k]) / 2) * ((U[i-1, j, k] + U[i, j, k]) / 2)

```

(Note parentheses.) Not only may this produce slightly different results due to differences in floating-point roundoff errors, but it in the `no_tmp!` case it is also likely to prevent the compiler from realizing that `upp == -ux1` and pulling out the common subexpression.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [June 5, 2024, 12:27pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/7 "2024-06-05T12:27:58Z")

</div>

I don’t get anywhere near 3x, but still a bigger difference than I would have expected:

```julia
julia> @btime plain!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG)
  56.065 ms (0 allocations: 0 bytes)

julia> @btime no_tmp!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG)
  69.160 ms (0 allocations: 0 bytes)

```

My CPU is quite similar to yours (`20 × 13th Gen Intel(R) Core(TM) i7-13800H`)

---

<div class="post-metadata">

**Author:** ![0samuraiE](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/0samuraie/32/209825_2.png) [@0samuraiE](https://discourse.julialang.org/u/0samuraiE)\
**Post date:** [June 5, 2024, 12:54pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/8 "2024-06-05T12:54:30Z")

</div>

Thank you all. I am surprised at such quick replies.  
I completely convinced by stevengj and understand reads should be done before writes. Also, float point operation difference is interesting.

But I still have question about 3x slower. And @inbounds did not fix this problem.  
I got 101.028 ms from the former and 324.322 ms from the latter.  
This is too slow comparing to nilshg.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 5, 2024, 1:52pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/9 "2024-06-05T13:52:47Z")

</div>

> [@0samuraiE](#):
>
> But I still have question about 3x slower.

`@benchmark` returns a lot of different statistics. Are you looking at the _minimum_ time? This is what `@btime` reports, and is often the most meaningful number.

---

<div class="post-metadata">

**Author:** ![0samuraiE](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/0samuraie/32/209825_2.png) [@0samuraiE](https://discourse.julialang.org/u/0samuraiE)\
**Post date:** [June 5, 2024, 2:05pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/10 "2024-06-05T14:05:49Z")

</div>

Thank you for answer. I saw probably mean time. All outputs are as follows.

```julia
julia> @benchmark plain!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG)
BenchmarkTools.Trial: 131 samples with 1 evaluation.
 Range (min … max): 35.522 ms … 58.969 ms ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 35.885 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 38.408 ms ± 5.696 ms ┊ GC (mean ± σ): 0.00% ± 0.00%

  █▇▂ ▁▁ ▁       
  ███▆██▄▄▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁██▄█▄▄ ▄
  35.5 ms Histogram: log(frequency) by time 52.4 ms <

 Memory estimate: 0 bytes, allocs estimate: 0.

julia> @benchmark noassign!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG)
BenchmarkTools.Trial: 57 samples with 1 evaluation.
 Range (min … max): 88.406 ms … 90.032 ms ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 88.995 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 89.011 ms ± 318.081 μs ┊ GC (mean ± σ): 0.00% ± 0.00%

                            █ ▁▁ ▁ ▁                    
  ▄▁▁▁▇▄▁▄▁▄▄▄▄▄▄▄▁▄▄▄▁▄▇▁▇▄█▄██▄▄▄█▄▇▄▁▁▇▄█▁▁▁▁▁▄▁▁▁▄▄▁▄▁▁▁▁▇ ▁
  88.4 ms Histogram: frequency by time 89.7 ms <

 Memory estimate: 0 bytes, allocs estimate: 0.

```

---

<div class="post-metadata">

**Author:** ![Seif\_Shebl](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@Seif\_Shebl](https://discourse.julialang.org/u/Seif_Shebl)\
**Post date:** [June 5, 2024, 5:48pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/11 "2024-06-05T17:48:16Z")

</div>

> [@0samuraiE](#):
>
> `U = rand!(zeros(N + 2NG, N + 2NG, N + 2NG));`

Aside from performance, why use `rand!(zeros(..))` and not just `rand(..)`? The latter would be more concise and faster.

---

<div class="post-metadata">

**Author:** ![Seif\_Shebl](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@Seif\_Shebl](https://discourse.julialang.org/u/Seif_Shebl)\
**Post date:** [June 5, 2024, 5:58pm UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/12 "2024-06-05T17:58:37Z")

</div>

In Julia version 1.11.0-DEV, I get identical timings (`--check-bounds=no` option is used),

```julia
BenchmarkTools.Trial: 69 samples with 1 evaluation.
 Range (min … max): 71.886 ms … 81.659 ms ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 72.572 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 73.115 ms ± 1.723 ms ┊ GC (mean ± σ): 0.00% ± 0.00%

    ▄█▅ ▂                                                      
  ▄▆███▃█▅▄▆▅▃▄▄▁▃▁▁▁▁▁▁▁▁▁▁▁▁▃▁▁▁▃▁▁▁▁▁▁▁▁▁▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▃ ▁
  71.9 ms Histogram: frequency by time 80.4 ms <

 Memory estimate: 0 bytes, allocs estimate: 0.
BenchmarkTools.Trial: 68 samples with 1 evaluation.
 Range (min … max): 72.288 ms … 96.033 ms ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 73.023 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 74.107 ms ± 4.257 ms ┊ GC (mean ± σ): 0.00% ± 0.00%

  ▅██▅                                                         
  ████▅▅▁▁▁▁▁▁▁▁▅▁▁▁▁▁▁▅▁▁▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▁▁▁▁▁▅ ▁
  72.3 ms Histogram: log(frequency) by time 92 ms <

 Memory estimate: 0 bytes, allocs estimate: 0.

```

---

<div class="post-metadata">

**Author:** ![0samuraiE](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/0samuraie/32/209825_2.png) [@0samuraiE](https://discourse.julialang.org/u/0samuraiE)\
**Post date:** [June 6, 2024, 1:37am UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/13 "2024-06-06T01:37:38Z")

</div>

Thank you for benchmarking. I agree rand! is a roundabout way.  
I also tried --check-bounds=no but same problem occurred.  
I’m convinced that’s the way it is.

Thank you all for answers. If someone gets same problem or knows the reason, please tell me.

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [June 6, 2024, 5:27am UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/14 "2024-06-06T05:27:02Z")

</div>

> [@0samuraiE](#):
>
> ```julia-repl
> julia> @benchmark noassign!($U, $V, $W, $Jx, $Jy, $Jz, $NX, $NY, $NZ, $NG)
> 
> ```

What’s `noassign!` here?

---

<div class="post-metadata">

**Author:** ![0samuraiE](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/0samuraie/32/209825_2.png) [@0samuraiE](https://discourse.julialang.org/u/0samuraiE)\
**Post date:** [June 7, 2024, 9:12am UTC](https://discourse.julialang.org/t/strange-performance-problem/115227/15 "2024-06-07T09:12:00Z")

</div>

I am sorry noassign! is the same function as no\_tmp!.