Just a heads up, this is in the works at MCMCTempering.jl. I had to put it on hold for a bit, but am about to return to it. Unfortunately it’s not quite ready for “public consumption” just yet, but it will be in not too long:)
And I’d expect tempering to be quite useful in this scenario!
There is also Pigeons.jl which is slightly more restrictive (though has fantastic support for distributed workloads), that should also work on Turing.jl: Turing.jl model · Pigeons.jl. Maybe give this a try?