I’m a newbie in julia language, I’ve written a parallel code using Distributed, and I want to write the intermediate results of the program’s calculations to a file, this is my code.
using Distributed
# Set up parallel workers
addprocs(4)
@everywhere begin
function my_function(n)
# Compute the result
result=n;
# Open the file in append mode
f = open("output.txt", "a")
# # Write the result to the file
println(f, "Result: ", result)
close(f)
# open("output.txt","a",lock=true) do io
# write(io,"Result: "*string(result))
# write(io,"\r\n")
# end
return result
end
end
@distributed for i in 1:10000
result = my_function(i)
println("Result for iteration $i: $result")
end
I want get output.txt as follow:
Result: 1
Result: 2501
Result: 2
Result: 3
Result: 4
Result: 5
Result: 6
Result: 7
Result: 8
Thank you very much, I will try the third option, the first two were a bit difficult for me.
In the documentation, I see this in the function open: “The lock keyword argument controls whether the operation will be locked to ensure secure multithreaded access.” But it doesn’t seem to work. Thanks again.
@johnh Thanks. Multithreading is really good for running on a single server. I tried it, and the result was the same, and the output file was also messy. Following Steward’s suggestion, I now use channels to log the data, and then a separate process writes the data.
I think there are two mistakes regarding the use of lock=true for open in the code above (which is the default by the way):
it’s meant for multithreading rather than multiprocess
it’s meant for multiple accesses to the io value returned by open
The second is the biggest mistake: if you do io = open(..., lock=true) in different threads then each thread will use its own io object with its own lock, so the locks are useless. Instead, you should call open only once, and share the io between threads.