I’m a newbie in julia language, I’ve written a parallel code using Distributed, and I want to write the intermediate results of the program’s calculations to a file, this is my code. using Distributed # Set up parallel workers addprocs(4) @everywhere begin function my_functi…

[image] zhpf0530: I’ve written a parallel code using Distributed, and I want to write the intermediate results of the program’s calculations to a file, this is my code. You are creating a race condition when multiple processes try to write to the same file simultaneously and end up overwritin…

Thank you very much, I will try the third option, the first two were a bit difficult for me. In the documentation, I see this in the function open: “The lock keyword argument controls whether the operation will be locked to ensure secure multithreaded access.” But it doesn’t seem to work. Thanks ag…

Given that your output is pretty simple, I think a fourth way to do this would be with Logging and then use LoggingExtras.jl to pipe them to a file.

@zhpf0530 I hope this is said in a nice manner. You are doing distributed processing. Multithreaded applies to running on a single server.

@johnh Thanks. Multithreading is really good for running on a single server. I tried it, and the result was the same, and the output file was also messy. Following Steward’s suggestion, I now use channels to log the data, and then a separate process writes the data.

I think there are two mistakes regarding the use of lock=true for open in the code above (which is the default by the way): it’s meant for multithreading rather than multiprocess it’s meant for multiple accesses to the io value returned by open The second is the biggest mistake: if you do io = op…

[image] zhpf0530: “The lock keyword argument controls whether the operation will be locked to ensure secure multithreaded access. You’re not doing multi-threaded access. You’re doing multi-process access. Google processes vs threads .

[image] zhpf0530: lock I have a case where I want to do something like what @zhpf0530 did. Instead, I am calling the @distributed macro from within a function that iterates over a collection of files and I do not need to use the @everywhere macro. My function is shown below, but I wonder if t…

[image] leoflotor: @distributed for file in files ... open(join([dir, savetofile]), "a", lock=true) ... lock=true only prevents races for multi-threaded access (single process with multiple threads, shared memory), not for distributed-memory access (multi-process, disjoint memory). T…

[image] stevengj: use a lock file to control access (so that only one process writes to at once), e.g. using Pidfile use a a different file format that supports parallel I/O (e.g. parallel HDF5 , though that requires you to use MPI). send all of the data you want to output to a single process,…

A problem: writing results to a file in parallel

New to Julia

jar1 December 15, 2023, 1:28am 15

I would categorize that under

3 Likes

Topic		Replies	Views
Save data to same files from independent, parallel processes Data parallel , input-output	1	91	November 21, 2024
Saving to a file during parallel computation General Usage	17	588	October 26, 2023
Atomic write(filename) append? General Usage	2	682	March 7, 2018
Write to text file while using threads New to Julia threads , io	12	1246	November 24, 2021
Parallelize simulations and print out results immediately when they are available New to Julia	4	887	May 9, 2019

A problem: writing results to a file in parallel

Related topics