I am trying to simulate a nested model that contains many for loops. I want to use the parallel calculation to speed up the simulation. But I failed to use @threads. Here is the structure of my code:
addprocs(10)
v_rep = 1:10
v_i = 1:10
v_j = 1:10
v_k = 1:10
@progress for rep = 1:10
println("rep = $rep")
val_rep = v_rep[rep]
Threads.@threads for (i, j, k) in collect(Iterators.product(1:3, 1:3, 1:3))
val_i = v_i[i]
val_j = v_j[j]
val_k = v_k[k]
val = [val_i,val_j,val_k]
# CSV.write("./val$rep$i$j$k.csv", val)
end
end
Often when asking for help with why something doesn’t work, it is helpful to include the stacktrace of whatever errors you get.
It is also helpful if code you share is self-contained (i.e., can be run simply by copy-paste in a new Julia session). For example, addprocs requires the Distributed standard library package to be loaded (i.e., using Distributed), and @progress requires, e.g., ProgressMeter.jl. When I removed addprocs(10) and @progress from your code, it ran without errors for me, so I’m not sure what problem you are having. (Or maybe loading those packages is the answer you need?)
As a side note, Threads.@threads uses the JULIA_NUM_THREADS environment variable to determine the number of threads Julia can use. Alternatively, one can start Julia with the -t option to specify the number of threads to use (e.g., julia -t 10). In particular, addprocs(10) spawns 10 Julia processes (that can be used for parallel work using the Distributed standard library package), but does not enable multithreading.
As a style (and somewhat performance) note, the call to collect is unnecessary, i.e., for (i, j, k) in Iterators.product(1:3, 1:3, 1:3) suffices (and usually (always?) is preferred). collect makes an Array, so it is only needed if you actually need an Array. Apparently collect is needed to work with @threads.
Interesting. I thought you’d have problems with the construction of the @threadsfor-loop but lo and behold the following is working for me on Julia 1.9.0 (and 1.7.2)
v_rep = 1:10
v_i = 1:10
v_j = 1:10
v_k = 1:10
for rep = 1:10
println("rep = $rep")
val_rep = v_rep[rep]
Threads.@threads for (i, j, k) in collect(Iterators.product(1:3, 1:3, 1:3))
val_i = v_i[i]
val_j = v_j[j]
val_k = v_k[k]
val = [val_i,val_j,val_k]
# CSV.write("./val$rep$i$j$k.csv", val)
end
end
(removed addprocs because that is necessary for Distributed, but not for Threads).
Edit: Ah, I think @lawless-m understood your problem. The incantation to fix this is
using Tables
CSV.write("./val$rep$i$j$k.csv", Tables.table(val))
P.S.: please mention the error that struck you next time…
Thanks for the detailed reply! This is my first time to ask question online. I will remember to include the errors.
I didn’t library the packages I needed. You are right. After I library the necessary packages, there is no error.
The code is simplified from a simulation model which contains many for loop. In front of each for, there always contains @process, which I actually don’t understand what it is doing. I direct copied and change the multiple and nested for loop to only one for loop, to catch the vector from a collect. I will remove @process in my original script and re-run it. Thank you!
julia -t auto should be from the command line, not in the REPL
You can also set the JULIA_NUM_THREADS environment variable, which I prefer because it is annoying to be part way through some work at the REPL and then want some threads available.
That @progress I removed, because I got the error message ERROR: LoadError: UndefVarError: @progress not defined. But it probably is Juno.@progress which shows a progress bar in the Atom/Juno IDE.
One thing that I noticed is that the threads are not being applied to the outer loop but instead to the inner loop. Is this a good thing, a bad thing, or does it not matter?