Controlling @threads (or similar) parallelism level

[Mason]

@evanfields

It’d be nice to something more like

memory_per_item = estimate_memory(chunk[1])
n_threads = floor(Int, system_memory / (memory_per_item * safe_inflation_factor))
Threads.@threads n_threads for i in 1:length(chunk)
    memory_intensive_computation(chunk[i])
end

i.e. dynamically choose the right parallelism level. What’s the best way to achieve this kind of pattern?

You can do this with OhMyThreads.jl

You’d just write

using OhMyThreads
memory_per_item = estimate_memory(chunk[1])

@tasks for i in eachindex(chunk)
    @set ntasks = floor(Int, system_memory / (memory_per_item * safe_inflation_factor))
    memory_intensive_computation(chunk[i])
end

or

using OhMyThreads
memory_per_item = estimate_memory(chunk[1])

ntasks = floor(Int, system_memory / (memory_per_item * safe_inflation_factor))

tforeach(eachindex(chunk); ntasks) do i
    memory_intensive_computation(chunk[i])
end