# Need help with performance in a very large loop

**URL:** <https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292>\
**Category:** Performance\
**Tags:** performance\
**Created:** [May 27, 2020, 9:06pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292 "2020-05-27T21:06:33Z")\
**Posts on this page:** 9\
**Page:** 2

<div class="post-metadata">

**Author:** ![jmcastro2109](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmcastro2109/32/38427_2.png) [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)\
**Post date:** [May 28, 2020, 3:55pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/21 "2020-05-28T15:55:10Z")

</div>

Thanks so much! Yes, regarding points (1) and (3) the code seems absurd because I abstracted away of some other intricacies of the code to build the MWE. Regarding point (2) I agree with you!

Thanks!

---

<div class="post-metadata">

**Author:** ![bcmichael](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bcmichael/32/10003_2.png) [@bcmichael](https://discourse.julialang.org/u/bcmichael)\
**Post date:** [May 28, 2020, 4:06pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/22 "2020-05-28T16:06:20Z")

</div>

So what is the actual difference between tempind1 and tempindWC? If they are different by a constant addition then you don’t need both even if that constant is different for each t.

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [May 28, 2020, 4:28pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/23 "2020-05-28T16:28:31Z")

</div>

> [@jmcastro2109](#):
>
> I abstracted away of some other intricacies of the code to build the MWE.

By, “MWE” do you mean the code posted at the top of this thread? It could be made much closer to minimal. You could get it to the point where it’s easy for people to run it. I wanted to try to run the code, but there are a lot of modules imported. I suspected they are not necessary. I don’t want to install them all; eg the excel file reader requires that I install other binaries. But, I don’t know which are necessary because you are importing all exportable symbols. So, to make it easier to help, I’d suggest something like the following, which imports only the modules that you actually use.

```julia
using Distributed
using BitOperations: BitOperations
using LinearAlgebra: LinearAlgebra

```

Then write `BitOperations.bget` and `LinearAlgebra.mul!`.

Also, you probably don’t want `@btime` but rather `@time` because you are timing something that takes a long, approximately constant, amont of time.

---

<div class="post-metadata">

**Author:** ![jmcastro2109](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmcastro2109/32/38427_2.png) [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)\
**Post date:** [May 28, 2020, 4:53pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/24 "2020-05-28T16:53:53Z")

</div>

The actual difference between temp1WC and temp1 is a vector of size 2^K that depends on the rows of ChoiceSet. I am writing an improvement on the current MWE to clarify these comments.

---

<div class="post-metadata">

**Author:** ![jmcastro2109](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmcastro2109/32/38427_2.png) [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)\
**Post date:** [May 28, 2020, 6:26pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/25 "2020-05-28T18:26:52Z")

</div>

I just wrote a cleaner MWE including your comments.

---

<div class="post-metadata">

**Author:** ![bcmichael](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bcmichael/32/10003_2.png) [@bcmichael](https://discourse.julialang.org/u/bcmichael)\
**Post date:** [May 28, 2020, 7:01pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/26 "2020-05-28T19:01:19Z")

</div>

I actually just found a big problem. You switched to using Float32 but best is still initialized with a Float64, which gives you a type instability in a hot loop. Replace `best = -Inf` with `best = -Inf32`, and do the same for bestWC etc.

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [May 28, 2020, 7:16pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/27 "2020-05-28T19:16:37Z")

</div>

> [@bcmichael](#):
>
> type instability in a hot loop

Fixing this decreases the run time for `k=7` by 65%.

---

<div class="post-metadata">

**Author:** ![jmcastro2109](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmcastro2109/32/38427_2.png) [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)\
**Post date:** [May 28, 2020, 7:19pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/28 "2020-05-28T19:19:11Z")

</div>

Thanks!!! Nice catch. This improves performance a lot!

---

<div class="post-metadata">

**Author:** ![bcmichael](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bcmichael/32/10003_2.png) [@bcmichael](https://discourse.julialang.org/u/bcmichael)\
**Post date:** [May 28, 2020, 10:58pm UTC](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292/29 "2020-05-28T22:58:47Z")

</div>

I don’t know if it’s the case or not in your real workload, but in this code you only use one of the values in Indices, but you calculate an index for every combination of i, iii, and t. You also calculate the index of bestWC, but don’t actually use it for anything. There is probably a speedup to be had by only calculating the values you actually need.

[Previous page](https://discourse.julialang.org/t/need-help-with-performance-in-a-very-large-loop/40292.md?page=1)
