# Allocations on multi threaded matrix operations (caused by BigFloats)

**URL:** <https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697>\
**Category:** General Usage\
**Tags:** multithreading, memory-allocation\
**Created:** [July 28, 2018, 8:29am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697 "2018-07-28T08:29:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 8:29am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/1 "2018-07-28T08:29:00Z")

</div>

I wrote a small test code that changes matrix elements and stores them in the “old” matrix, so no new allocations should be made.  
I then compiled the code with @allocated to measure the allocations made:

```julia
        - function test(N)
    80080 M = Array{BigFloat}(N,N)
        0 for i = 1:N
        0 for j = 1:N
        0 M[i,j] = big(pi)	#spawn matrix elements
        - end
        - end
        - 
        - #Now change matrix elements
        0 for i = 1:N
        0 for j = 1:N
        0 M[i,j] = BigFloat(2)*M[i,j]
        - end
        - end
        - end
        - 

```

As you can see, the code behaves as expected with only allocations made when the matrix is initialized.

However, when I add multi threading, new allocations are being made:

```julia
        - function test(N)
    80096 M = Array{BigFloat}(N,N)
        0 Threads.@threads for i = 1:N
     1176 Threads.@threads for j = 1:N
   174744 M[i,j] = big(pi)	#spawn matrix elements
        - end
        - end
        - 
        - #Now change matrix elements
        0 Threads.@threads for i = 1:N
     1640 Threads.@threads for j = 1:N
   773132 M[i,j] = BigFloat(2)*M[i,j]
        - end
        - end
        - end
        - 

```

Can anyone explain this behavior and tell me how to bypass it?

Thanks in advance

EDIT:  
To give you a quick update:

I tried to make my algorithm work by only using Arbs, to avoid getting new allocations by the BigFloats.

Unfortunately, this does not work because Arbs seem to not be made for this kind of usage.  
They lose numerically stability during the matrix operations and get unprecise results (unless one puts the precision much higher than for BigFloats, but this makes the performance even worse than the BigFloat-allocations).

I would like to know if someone knows another way how I could make this type of algorithm work.

Maybe there is another multiprecision floating type or a way to preallocate the BigFloat calculations?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 28, 2018, 9:47am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/2 "2018-07-28T09:47:55Z")

</div>

Not directly related:  
Nesting threaded for loops,  
is not valid in julia 0.6 and can lead to incorrect results.  
In 0.7 it is valid but it does nothing for the inner loop.  
Removing it from the inner loop might save some allocations.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 28, 2018, 9:53am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/3 "2018-07-28T09:53:44Z")

</div>

Just looking at the allocations as reported by `@time`  
in julia 0.6,  
I don’t see any significant difference in number of allocations between the threaded versions and the nonthreaded versions.

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 10:54am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/4 "2018-07-28T10:54:48Z")

</div>

Yes, you are right, I did that correct in the real code that I am working with but forgot to do it in the example.

```julia
        - function test(N)
    80096 M = Array{BigFloat}(N,N)
        0 Threads.@threads for i = 1:N
        0 for j = 1:N
        0 M[i,j] = big(pi)	#spawn matrix elements
        - end
        - end
        - 
        - #Now change matrix elements
        0 Threads.@threads for i = 1:N
        0 for j = 1:N
  1113860 M[i,j] = BigFloat(2)*M[i,j]
        - end
        - end
        - end
        - 

```

The second loop still shows the same behavior tho

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 10:56am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/5 "2018-07-28T10:56:14Z")

</div>

@time shows allocations for me too:

```julia
julia> @time test(100)
  0.090719 seconds (106.77 k allocations: 3.996 MiB)

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [July 28, 2018, 10:57am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/6 "2018-07-28T10:57:50Z")

</div>

Creating `BigFloat` allocates.

Ref [WIP: Pooling BigInts and BigFloats by skariel · Pull Request #10084 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/10084)

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 11:00am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/7 "2018-07-28T11:00:52Z")

</div>

Oh, is there a way to avoid this when one is forced to work with BigFloats?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [July 28, 2018, 11:03am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/8 "2018-07-28T11:03:50Z")

</div>

Perhaps @JeffreySarnoff knows some good alternatives?

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [July 28, 2018, 11:22am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/9 "2018-07-28T11:22:44Z")

</div>

Do you really need to be using BigFloats?

```julia
julia> using DoubleFloats
julia> sqrt(Double64(2))
1.4142135623730950488016887242096816
jjulia> inv(Double64(MathConstants.golden))
6.1803398874989484820458683436563545e-01
julia> exp(ans) * cbrt(ans)
1.5803242493055311416008395194148359

```

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 11:29am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/10 "2018-07-28T11:29:09Z")

</div>

Yes, I have to calculate the determinant of an ill conditioned 2000x2000 matrix up to 2400 digits precision ☹

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [July 28, 2018, 11:30am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/11 "2018-07-28T11:30:07Z")

</div>

Have you tried ArbFloats.jl?

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 11:31am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/12 "2018-07-28T11:31:26Z")

</div>

I have already implemented them because MPFR does not provide multi precision besselj functions of fractional order. I will try how it works if I change the matrix to this type.

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [July 28, 2018, 11:33am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/13 "2018-07-28T11:33:04Z")

</div>

Just change the type of the numbers in the matrix, not the matrix itself.  
I have not provided Bessel functions in that package.

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 11:39am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/14 "2018-07-28T11:39:09Z")

</div>

No, I am working with Nemo, but these are ArbFloats too, right?

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [July 28, 2018, 11:39am UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/15 "2018-07-28T11:39:31Z")

</div>

correct

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 12:10pm UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/16 "2018-07-28T12:10:20Z")

</div>

That works much better:

```julia
        - using ArbFloats
        - function test(N)
    80096 M = Array{ArbFloat}(N,N)
        0 Threads.@threads for i = 1:N
        0 for j = 1:N
        0 M[i,j] = ArbFloat(pi)	#spawn matrix elements
        - end
        - end
        - 
        - #Now change matrix elements
        0 Threads.@threads for i = 1:N
        0 for j = 1:N
    31458 M[i,j] =2*M[i,j]
        - end
        - end
        - end
        - 

```

Thank you for the suggestion!

On the website it said “Much faster than BigFloat at precisions up to 3,500 bits (1050 digits)”  
If I reach more than that, is the speed comparable to BigFloats or does it become extremely slow?

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [July 28, 2018, 12:24pm UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/17 "2018-07-28T12:24:13Z")

</div>

worry not – it will not become slower than BigFloats the way you will use it.

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 12:30pm UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/18 "2018-07-28T12:30:25Z")

</div>

Okay great, and I will save the time of the allocations too!

---

<div class="post-metadata">

**Author:** ![favba](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/favba/32/2735_2.png) [@favba](https://discourse.julialang.org/u/favba)\
**Post date:** [July 28, 2018, 12:37pm UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/19 "2018-07-28T12:37:51Z")

</div>

As a side note, you will have better performance if you change the nesting order of the loops to

```julia
for j= ...
    for i= ...
        M[i,j] = ...

```

Julia arrays are stored in Fortran order.

---

<div class="post-metadata">

**Author:** ![DavidBerghaus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidberghaus/32/4623_2.png) [@DavidBerghaus](https://discourse.julialang.org/u/DavidBerghaus)\
**Post date:** [July 28, 2018, 12:38pm UTC](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697/20 "2018-07-28T12:38:24Z")

</div>

Is there a way to convert Nemo.arb to ArbFloats.ArbFloat?  
I need to multiply these two types and it gives me errors when I do that.

[Next page](https://discourse.julialang.org/t/allocations-on-multi-threaded-matrix-operations-caused-by-bigfloats/12697.md?page=2)