# Thread Local Storage in Julia

**URL:** <https://discourse.julialang.org/t/thread-local-storage-in-julia/11765>\
**Category:** Internals & Design\
**Tags:** proposal\
**Created:** [June 18, 2018, 8:36pm UTC](https://discourse.julialang.org/t/thread-local-storage-in-julia/11765 "2018-06-18T20:36:54Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gaetano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gaetano/32/4654_2.png) [@Gaetano](https://discourse.julialang.org/u/Gaetano)\
**Post date:** [June 18, 2018, 8:36pm UTC](https://discourse.julialang.org/t/thread-local-storage-in-julia/11765/1 "2018-06-18T20:36:55Z")

</div>

Lately I begin to tackle the RNG thread safety issue, you can find the full story [here](https://dokfaust.github.io/2018/06/16/julia-rng.html).

After a couple of uncessfull attempts, I have check out how Rust has implemented a [multi-thread RNG generator](https://docs.rs/rand/0.4.2/src/rand/lib.rs.html#879-893). It functions in a Local Key fashion, like Thread -Specific data in pthread.

I was thinking about writing a Julia wrapper to `pthread_setspecific()` but it seems that it would against the premise of `getptls()` that there is a unique pthread key, as each user allocated value would require a static key.

The point is, has somebody ever had the necessity of TLS? Because if not, writing a general wrapper would be unnecessary.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 19, 2018, 12:56pm UTC](https://discourse.julialang.org/t/thread-local-storage-in-julia/11765/2 "2018-06-19T12:56:12Z")

</div>

See also: [https://github.com/JuliaLang/julia/issues/10441](https://github.com/JuliaLang/julia/issues/10441) … it is explained there (and in [https://github.com/JuliaLang/julia/pull/26562](https://github.com/JuliaLang/julia/pull/26562)) how to get thread-local storage of the RNG.

Two difficulties: how to do this by default without degrading single-thread performance, and what parallel RNG algorithm to use in order to get statistically independent streams.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 19, 2018, 1:03pm UTC](https://discourse.julialang.org/t/thread-local-storage-in-julia/11765/3 "2018-06-19T13:03:46Z")

</div>

As Jeff commented in one of the issues I linked, thread local storage is much more efficient than task local for this kind of thing, mainly because Julia uses a fixed pool of threads created when Julia starts.

---

<div class="post-metadata">

**Author:** ![Gaetano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gaetano/32/4654_2.png) [@Gaetano](https://discourse.julialang.org/u/Gaetano)\
**Post date:** [June 19, 2018, 9:44pm UTC](https://discourse.julialang.org/t/thread-local-storage-in-julia/11765/4 "2018-06-19T21:44:11Z")

</div>

Hi @stevengj, thanks for you response! I tried to write a smal script to test a similar version of a RNG in a thread indexed array :

```julia
using Random, BenchmarkTools

function init_TRNG(rng = MersenneTwister(0), gpt = 1)
    n = Threads.nthreads()
    rngjmp = randjump(rng, big(10)^20, n*(gpt+1))
    threadRNG = reshape(rngjmp, (gpt+1,n))[1:gpt,:]
    return threadRNG
end

TRNG = init_TRNG()

trand() = rand(@inbounds(TRNG[Threads.threadid()]))

function f(n::Int)
    a = zeros(n)
    Threads.@threads for i in 1:n
        a[i] = rand()
    end
    return a
end

function g(n::Int)
    a = zeros(n)
    Threads.@threads for i in 1:n
        a[i] = trand()
    end
    return a
end
f(400)
g(400)

```

Using the [perf patch](https://github.com/JuliaLang/julia/pull/27466) I have noticed that `g()` spends more time in `ti_threadgroup_fork` (called by `@threads`) during thread synchronization.

This gives a 5-10% overhead, also it seems that `randjump` is quite slow and this affects seriously benchmarks, especially for small loop cycles.

Also the most interesting part is that the extra `threadid()` call gives just a \<1% overhead, so probably any attempt to circumvent it won’t be significative. (eg pthread TLS initialization or task local generators).

Regarding the two difficulties :

1. What about having two separate functions?
2. Using `randjump` gives independent streams. Yeah maybe finding a faster solution could be interesting.

Thanks again for your reply!

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 20, 2018, 3:29am UTC](https://discourse.julialang.org/t/thread-local-storage-in-julia/11765/5 "2018-06-20T03:29:33Z")

</div>

> [@Gaetano](#):
>
> `TRNG = init_TRNG()`

Don’t use non-constant globals, they will kill performance. Do e.g.

```julia
const TRNG = [MersenneTwister(0) for i = 1:Threads.nthreads()]
function trand()
    @inbounds rng = TRNG[Threads.threadid()]
    return rand(rng)
end

```

Using BenchmarkTools, I find that `@btime rand()` takes 5.7ns on my machine, whereas `@btime trand()` takes 10.6ns, a fairly significant slowdown unfortunately (this is with Julia 0.6.3). (This is the sort of thing that might benefit from compiler support for thread-local variables, maybe.)

On the other hand, the overhead will be much less if you do something like `rand!(array)`, in which the overhead of looking up the RNG only occurs once.

---

<div class="post-metadata">

**Author:** ![Gaetano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gaetano/32/4654_2.png) [@Gaetano](https://discourse.julialang.org/u/Gaetano)\
**Post date:** [June 20, 2018, 10:30pm UTC](https://discourse.julialang.org/t/thread-local-storage-in-julia/11765/6 "2018-06-20T22:30:06Z")

</div>

> [@stevengj](#):
>
> Don’t use non-constant globals, they will kill performance. Do e.g.

Doh! Sorry yeah that was penzalizing my benchmarks.

> [@stevengj](#):
>
> Using BenchmarkTools, I find that `@btime rand()` takes 5.7ns on my machine, whereas `@btime trand()` takes 10.6ns,

Don’t you ha ve a 0.7 build? Because initializing TRNG with `randjump` I get :

```julia
@btime trand()
5.057 ns
@btime rand()
4.707 ns

```

which seems more acceptable!  
Also the interesting part comes from benchmarks

```julia
julia> @benchmark g(400)
BenchmarkTools.Trial:
  memory estimate: 3.28 KiB
  allocs estimate: 2
  --------------
  minimum time: 3.276 μs (0.00% GC)
  median time: 3.741 μs (0.00% GC)
  mean time: 9.390 μs (43.47% GC)
  maximum time: 34.850 ms (99.85% GC)
  --------------
  samples: 10000
  evals/sample: 8

julia> @benchmark f(400)
BenchmarkTools.Trial:
  memory estimate: 3.28 KiB
  allocs estimate: 2
  --------------
  minimum time: 6.676 μs (0.00% GC)
  median time: 7.487 μs (0.00% GC)
  mean time: 19.037 μs (43.10% GC)
  maximum time: 73.441 ms (99.32% GC)
  --------------
  samples: 10000
  evals/sample: 4

```

so that the `trand()` loop in `g` is twice as fast!  
This seems even an improvement on performance!