# \[ANN\] LoopVectorization
**URL:** https://discourse.julialang.org/t/ann-loopvectorization/32843
**Category:** Package Announcements
**Created:** [January 1, 2020, 8:08am UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843 "2020-01-01T08:08:57Z")
**Posts on this page:** 20
**Page:** 7
### Author:  [@Elrod](https://discourse.julialang.org/u/Elrod)
#### Post date: [May 25, 2020, 9:10am UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/123 "2020-05-25T09:10:47Z")
I’ve released LoopVectorization `v0.8.1` today (and `v0.8.0` yesterday).
The `0.8`-series introduced a breaking change:
It now assumes that none of the iterables are empty. That means `for i in 1:0; end` is invalid, but `for I in CartesianIndices(()); end` is still fine.
```julia
julia> isempty(1:0)
true
julia> isempty(CartesianIndices(()))
false
```
If you want LoopVectorization to emit a check, you can pass `check_empty=true`, and it will only run the code if the iterables are in fact not empty.
```julia
julia> using LoopVectorization, Test
julia> function mysum_checked(x)
s = zero(eltype(x))
@avx check_empty=true for i ∈ eachindex(x)
s += x[i]
end
s
end
mysum_checked (generic function with 1 method)
julia> function mysum_unchecked(x)
s = zero(eltype(x))
@avx for i ∈ eachindex(x)
s += x[i]
end
s
end
mysum_unchecked (generic function with 1 method)
julia> x = fill(9999, 100, 10, 10);
julia> @test mysum_checked(x) == mysum_unchecked(x) == sum(x)
Test Passed
julia> xv = view(x, :, 1:0, :);
julia> @test iszero(mysum_checked(xv))
Test Passed
julia> @test iszero(mysum_unchecked(xv))
Test Failed at REPL[26]:1
Expression: iszero(mysum_unchecked(xv))
ERROR: There was an error during testing
```
Maybe checking should be the default, as it takes extremely little time. But on Zulip, a few folks thought no-checks were reasonable.
LoopVectorization 0.8 had a major overhaul on how the resulting expression is emitted, with the focus on improving both performance and the quality of the generated assembly.
Matrix-multiplication performance is now very good:
While you’d still need to add extra loops and pack for a real matrix multiplication function, the raw macrokernel does extremely well on my system with AVX512.
[Brief aside: If you have any idea how to make all the colors easier to tell apart, please chime in on [this issue](https://github.com/chriselrod/LoopVectorization.jl/issues/107) or in this thread.]
I may have to tune it a bit for AVX2. I haven’t tested it there, but I think I am currently underestimating a cost for those architectures. Were I to make it more accurate, it may make a different decision that might improve performance there. “May”, “might” – I don’t know until I test.
Improvements in the assembly include eliminating a lot of redundant blocks and unnecessary addressing calculations.
Sample of the generated assembly on Julia master with LLVM 10 (disclaimer: there are still a few unnecessary instructions with LLVM 6, 8, and 9; I didn’t check 7 because Julia skipped from 6 to 8 with 1.3 → 1.4):
```plaintext
L1024:
prefetcht0 byte ptr [r11 + r14]
vbroadcastsd zmm30, qword ptr [r8]
vmovups zmm29, zmmword ptr [r11] vmovups zmm28, zmmword ptr [r11 + 64]
vmovupd zmm27, zmmword ptr [r11 + 128] prefetcht0 byte ptr [r11 + r14 + 64]
prefetcht0 byte ptr [r11 + r14 + 128]
vfmadd231pd zmm26, zmm30, zmm29 # zmm26 = (zmm30 * zmm29) + zmm26
vfmadd231pd zmm23, zmm30, zmm28 # zmm23 = (zmm30 * zmm28) + zmm23
vbroadcastsd zmm31, qword ptr [r8 + rdi]
vfmadd231pd zmm17, zmm30, zmm27 # zmm17 = (zmm30 * zmm27) + zmm17
vfmadd231pd zmm25, zmm31, zmm29 # zmm25 = (zmm31 * zmm29) + zmm25
vfmadd231pd zmm21, zmm31, zmm28 # zmm21 = (zmm31 * zmm28) + zmm21
vfmadd231pd zmm14, zmm31, zmm27 # zmm14 = (zmm31 * zmm27) + zmm14
vbroadcastsd zmm30, qword ptr [r8 + 2*rdi]
vfmadd231pd zmm24, zmm30, zmm29 # zmm24 = (zmm30 * zmm29) + zmm24
vfmadd231pd zmm19, zmm30, zmm28 # zmm19 = (zmm30 * zmm28) + zmm19
vfmadd231pd zmm11, zmm30, zmm27 # zmm11 = (zmm30 * zmm27) + zmm11
vbroadcastsd zmm30, qword ptr [r8 + r9]
vfmadd231pd zmm22, zmm30, zmm29 # zmm22 = (zmm30 * zmm29) + zmm22
vfmadd231pd zmm16, zmm30, zmm28 # zmm16 = (zmm30 * zmm28) + zmm16
vfmadd231pd zmm8, zmm30, zmm27 # zmm8 = (zmm30 * zmm27) + zmm8
vbroadcastsd zmm30, qword ptr [r8 + 4*rdi]
vfmadd231pd zmm20, zmm30, zmm29 # zmm20 = (zmm30 * zmm29) + zmm20
vfmadd231pd zmm13, zmm30, zmm28 # zmm13 = (zmm30 * zmm28) + zmm13
vfmadd231pd zmm6, zmm30, zmm27 # zmm6 = (zmm30 * zmm27) + zmm6
vbroadcastsd zmm30, qword ptr [r8 + r15]
vfmadd231pd zmm18, zmm30, zmm29 # zmm18 = (zmm30 * zmm29) + zmm18
vfmadd231pd zmm10, zmm30, zmm28 # zmm10 = (zmm30 * zmm28) + zmm10
vfmadd231pd zmm4, zmm30, zmm27 # zmm4 = (zmm30 * zmm27) + zmm4
vbroadcastsd zmm30, qword ptr [r8 + r12]
vfmadd231pd zmm15, zmm30, zmm29 # zmm15 = (zmm30 * zmm29) + zmm15
vfmadd231pd zmm7, zmm30, zmm28 # zmm7 = (zmm30 * zmm28) + zmm7
vbroadcastsd zmm31, qword ptr [r8 + rbp]
vfmadd231pd zmm2, zmm30, zmm27 # zmm2 = (zmm30 * zmm27) + zmm2
vfmadd231pd zmm12, zmm31, zmm29 # zmm12 = (zmm31 * zmm29) + zmm12
vfmadd231pd zmm5, zmm31, zmm28 # zmm5 = (zmm31 * zmm28) + zmm5
vfmadd231pd zmm1, zmm31, zmm27 # zmm1 = (zmm31 * zmm27) + zmm1
vbroadcastsd zmm30, qword ptr [r8 + 8*rdi]
vfmadd231pd zmm9, zmm30, zmm29 # zmm9 = (zmm30 * zmm29) + zmm9
vfmadd231pd zmm3, zmm30, zmm28 # zmm3 = (zmm30 * zmm28) + zmm3
vfmadd231pd zmm0, zmm30, zmm27 # zmm0 = (zmm30 * zmm27) + zmm0
add r11, r10
add r8, 8
cmp r11, rdx
jbe L1024
```
To see how far this has come, compare the above graphic with the opening post!
---
### Author:  [@DNF](https://discourse.julialang.org/u/DNF)
#### Post date: [May 25, 2020, 9:27am UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/124 "2020-05-25T09:27:56Z")
> [@Elrod](#):
>
> [Brief aside: If you have any idea how to make all the colors easier to tell apart, please chime in on [this issue](https://github.com/chriselrod/LoopVectorization.jl/issues/107) or in this thread.]
I don’t know any direct way in Julia, but some dude made a tool for Matlab some years ago, called “Maximally perceptually distinct colors” for Matlab plots. Maybe it could be ported to Julia?[Generate maximally perceptually-distinct colors - File Exchange - MATLAB Central](https://se.mathworks.com/matlabcentral/fileexchange/29702-generate-maximally-perceptually-distinct-colors?s_tid=prof_contriblnk)
I agree that the colors are a bit tough to distinguish.
---
### Author:  [@yakir12](https://discourse.julialang.org/u/yakir12)
#### Post date: [May 25, 2020, 9:29am UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/125 "2020-05-25T09:29:24Z")
This works really well:
[http://juliagraphics.github.io/Colors.jl/stable/colormapsandcolorscales/#Generating-distinguishable-colors-1](http://juliagraphics.github.io/Colors.jl/stable/colormapsandcolorscales/#Generating-distinguishable-colors-1)
---
### Author:  [@DNF](https://discourse.julialang.org/u/DNF)
#### Post date: [May 25, 2020, 9:34am UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/126 "2020-05-25T09:34:26Z")
Ah, I would guess that these two are connected 😃
---
### Author:  [@Elrod](https://discourse.julialang.org/u/Elrod)
#### Post date: [May 25, 2020, 9:40am UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/127 "2020-05-25T09:40:00Z")
[I had tried that](https://github.com/chriselrod/LoopVectorization.jl/blob/b09005245452de1f72b1f0fca59a2bc05bc9a6a5/benchmark/plotbenchmarks.jl#L17), but apparently I thought what I have now looked better.
Maybe [this is what that looked like](https://github.com/chriselrod/LoopVectorization.jl/blob/0440ba05cdb63697071ef2bf81706ede9dd27f5a/docs/src/assets/bench_AmulB_v1.png)?
Now I’m not so sure.
---
### Author:  [@jtackm](https://discourse.julialang.org/u/jtackm)
#### Post date: [May 25, 2020, 10:16am UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/128 "2020-05-25T10:16:35Z")
Looks better to me, but still heavy on the blues. Some traces are almost identical (e.g. ifort/ifort-builtin, icc/GFortran if I read correctly), aggregating them should already help color distinction. You can always provide detailed traces in a separate plot for those interested in these details.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [May 25, 2020, 1:05pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/129 "2020-05-25T13:05:04Z")
Thanks, it’s looking great! I think we’re seeing some installation issues with LoopVectorization on weird CPUs / platforms, but instead of just giving up on LoopVectorization, I hope you’re okay with us just pinging you each time we identify that it could be due to LV. I want to help make sure we can get this stable enough that we can use it without DiffEq, and then just bail on the weird cases, because this kind of performance leg-up is exactly the kind of thing we always want more of.
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [May 25, 2020, 1:24pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/130 "2020-05-25T13:24:38Z")
> [@ChrisRackauckas](#):
>
> because this kind of performance leg-up is exactly the kind of thing we always want more of.
I was not going to bring this up but since it seems that people actually want to use this in real packages, I just want to point out that,
1. All of the optimizations mentioned in [`README`](https://github.com/chriselrod/LoopVectorization.jl/tree/b09005245452de1f72b1f0fca59a2bc05bc9a6a5#examples) are actually handled by LLVM and some of the descriptions there are fairly misleading. In fact, LLVM’s issue is that it is too aggressive and the fallback isn’t gental enough.
2. This is really not the right level to do auto-vectorization. LLVM has so much better understanding of what’s going on and can take advantage of that much better. Hand vectorization, like SIMD.jl, is completely different and isn’t a problem.
Now with the current implementation of LLVM, you are certainly free to bypass it’s vectorizer to solve any immediate problems. However, don’t expect this to be a long term solution…
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [May 25, 2020, 1:33pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/131 "2020-05-25T13:33:36Z")
> [@yuyichao](#):
>
> Now with the current implementation of LLVM, you are certainly free to bypass it’s vectorizer to solve any immediate problems. However, don’t expect this to be a long term solution…
Right now, short term solutions are much better than long term solutions.
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [May 25, 2020, 1:35pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/132 "2020-05-25T13:35:13Z")
Another clue that could be used is line thickness.
---
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [May 25, 2020, 1:56pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/133 "2020-05-25T13:56:54Z")
…and/or line-styles (dotted, dashed, dash-dotted). You could also use the same colour for the three ifort lines but with different styles, you immediate gain two more colours.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [May 25, 2020, 2:13pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/134 "2020-05-25T14:13:24Z")
> [@yuyichao](#):
>
> I was not going to bring this up but since it seems that people actually want to use this in real packages, I just want to point out that,
>
> 1. All of the optimizations mentioned in [`README`](https://github.com/chriselrod/LoopVectorization.jl/tree/b09005245452de1f72b1f0fca59a2bc05bc9a6a5#examples) are actually handled by LLVM and some of the descriptions there are fairly misleading. In fact, LLVM’s issue is that it is too aggressive and the fallback isn’t gental enough.
> 2. This is really not the right level to do auto-vectorization. LLVM has so much better understanding of what’s going on and can take advantage of that much better. Hand vectorization, like SIMD.jl, is completely different and isn’t a problem.
>
> Now with the current implementation of LLVM, you are certainly free to bypass it’s vectorizer to solve any immediate problems. However, don’t expect this to be a long term solution…
This kind of comment is quite unfortunate and demoralizing. A lot of people have seen tremendous performance benefits from using LoopVectorization and I sure hope we continue to see this excellent work. If you have some ideas on how to improve upon things, I’m sure constructive criticism will be appreciated.
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [May 25, 2020, 2:32pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/135 "2020-05-25T14:32:17Z")
> [@baggepinnen](#):
>
> If you have some ideas on how to improve upon things, I’m sure constructive criticism will be appreciated.
Yes I do
> [@yuyichao](#):
>
> - In fact, LLVM’s issue is that it is too aggressive and the fallback isn’t gental enough.
And I didn’t give too detail comment and even mention that it is certainly OK to use it now exactly because I understand that a workaround for a problem is OK to have.
> [@baggepinnen](#):
>
> demoralizing
And that’s to a certain degree exactly the point, i.e. to discourage people from going down the wrong path too deep.
A workaround is a workaround. The correct level for this to happen is much lower so it wouldn’t be wise to encourage people to sink more time into it than what’s necessary to workaround existing issues since the workaround has it’s intrinsic limitations. (There’s no way it can know memory access and tradeoffs as much as LLVM do, without basically reimplementing the whole compiler infrastructure).
Also, the point is to not let the wrong information around mislead people to think this is what it really isn’t.
* * *
And if you do want more detailed comment.
> For this reason, the `@avx` version is roughly twice as fast. The `@inbounds @simd` version, however, is not, because it runs into the problem of loop carried dependencies: to add `a[i]*b[i]` to `s_new = s_old + a[i-j]*b[i-j]` , we must have first finished calculating `s_new` , but – while two `fma` instructions can be initiated per cycle – they each take several clock cycles to complete. For this reason, we need to unroll the operation to run several independent instances concurrently. The `@avx` macro models this cost to try and pick an optimal unroll factor.
Nop, LLVM does it too. In fact, from the benchmark above, it’s clear that LLVM only do badly for odd element sizes and is actually doing somewhat better for sizes that doesn’t need scaler loop. That’s because LLVM has a better model but doesn’t have a good handling for the tail.
[GitHub - JuliaSIMD/LoopVectorization.jl at b09005245452de1f72b1f0fca59a2bc05bc9a6a5](https://github.com/chriselrod/LoopVectorization.jl/tree/b09005245452de1f72b1f0fca59a2bc05bc9a6a5#matrix-multiply) appears to be the same edge effect. Again, it’s real, but the gain isn’t from the claimed reason above.
The syntax trick for broadcast is nice, but not particularly tied to vectorization…
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [May 25, 2020, 2:39pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/136 "2020-05-25T14:39:26Z")
And again, I have no problem people uses this as long as they are aware of it’s intrinsic limitations and know where the fix would be and don’t be too surprised in the future. And I believe I’m qualified to say this because I even made [GitHub - yuyichao/FunctionWrappers.jl](https://github.com/yuyichao/FunctionWrappers.jl) and it’s even written in the README that it is just a proof of principle.
---
### Author:  [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)
#### Post date: [May 25, 2020, 2:40pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/137 "2020-05-25T14:40:58Z")
Is the problem because Julia doesn’t expose enough information to LLVM or because LLVM doesn’t use it? If the latter, is there any reason to expect LLVM (which by now is a pretty mature piece of software) will get around to it?
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [May 25, 2020, 2:46pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/138 "2020-05-25T14:46:40Z")
No it’s because LLVM’s optimizer optimized deeply for the vectorized loop (apparently better than the package) but not for the rest of the element that it can’t handle. When you are talking about each loop iteration going through 32 elements in a optimized vector loop, the (order of) 31 element unoptimized scalar loop is a huge hit for dimensions as small as 255 (dot product example) or 70-80 (matrix multiply example). The scalar loop can easily take as much or longer time than the vectorized loop while processing fewer elements in those cases.
> [@antoine-levitt](#):
>
> If the latter, is there any reason to expect LLVM (which by now is a pretty mature piece of software) will get around to it?
= = if it is mature there won’t be much LLVM vs GCC (I still frequently see cases where GCC compiles much better than Clang, at least as often than the other way around…) or API breaking …
It’s a pure LLVM internal fix and I do expect it to happen as long as someone reports it. If anyone is intersted, a good way to do so would be to reproduce in clang (fairly easy) and to show what should the generated code be by using the proper compiler intrinsics, say, by essentially disassemble/reverse compile the asm/llvm ir generated by this package. More tweak would be needed on top of that for sure but an example of better codegen should get it started…
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [May 25, 2020, 2:58pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/139 "2020-05-25T14:58:11Z")
But wouldn’t then the solution be unavailable until Julia catches up to the appropriate LLVM version? At this point we seem to be a couple of versions behind the current…
---
### Author:  [@Ratingulate](https://discourse.julialang.org/u/Ratingulate)
#### Post date: [May 25, 2020, 3:15pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/140 "2020-05-25T15:15:27Z")
can you articulate exactly what the substantial fix would be and how hard that would be to implement?
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [May 25, 2020, 3:16pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/141 "2020-05-25T15:16:11Z")
No, we carry many patches.
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [May 25, 2020, 3:17pm UTC](https://discourse.julialang.org/t/ann-loopvectorization/32843/142 "2020-05-25T15:17:49Z")
Not really.
It shouldn’t be hard for people that have worked on the llvm vectorization patches but I don’t know all the detail there. This is also the reason I didn’t want to say anything initialially before people got too excited…
[Previous page](https://discourse.julialang.org/t/ann-loopvectorization/32843.md?page=6)
[Next page](https://discourse.julialang.org/t/ann-loopvectorization/32843.md?page=8)