Parallelism within Turing.jl model

Specific Domains Probabilistic Programming
, ,
7

As a Turing.jl developer in 2025 I’d just like to give a bit of an update on this, particularly for anybody who comes here via Google.

The official line is: Threads.@threads still works in Turing models, but it only works for likelihood terms i.e. x ~ dist where x is observed data (either specified in model arguments or conditioned upon). If you attempt to use it with prior terms, i.e. x is a random variable, the behaviour is undefined; last time I looked into it, doing this will either error or give wrong results.

Now, my (strong) personal opinion is that Threads.@threads, as well as the code in Turing that exists to handle this gracefully (DynamicPPL.ThreadSafeVarInfo for those in the know), is an antipattern. For example, see this Julia blog post. I would like to get rid of all of it, but I don’t yet know how to do it in a nice way. If you’re interested, you can see this issue and this PR.

It is a bit less “friendly” and more manual, but for those who are comfortable with Julia / Turing, I would like to gently suggest that instead of using

Threads.@threads for i in eachindex(x)
    some_other_computation()
    x[i] ~ dist
end

you instead explicitly calculate the log-likelihoods using logpdf, and then add the cumulative likelihood using @addlogprob! (see docs)

loglikes = map(x) do xi
    Threads.@spawn begin
        some_other_computation()
        return logpdf(dist, xi)
    end
end
@addlogprob! sum(fetch.(loglikes))

For Turing users, there are several benefits to this latter approach.

  1. You guard against future changes to the way parallelism works in Julia / Turing. As explained in the linked issue above, the threadsafe handling code in Turing is hacky and only happens to work in practice because we don’t typically run into the threadid issues described here. If Julia’s behaviour changes, we will have to change things in Turing accordingly, which may end up breaking Threads.@threads.
  2. Sampling a model that uses Threads.@spawn together with MCMCThreads() will yield reproducible results when the RNG is seeded. On the other hand, the combination of Threads.@threads and MCMCThreads() is not reproducible, even when seeding the RNG.
  3. Instead of directly using Threads.@spawn you can also use any parallelism library of your choice to calculate the likelihoods. (See subsequent comments in this thread for examples of OhMyThreads.jl and FLoops.jl.)
  4. You can also more easily extract the results of some_other_computation() in a thread-safe manner, should you need it outside the threaded loop: just add it to the return value of the Threads.@spawn block.
1 Like