Using locks in pmap

General Usage
1

I have an @everywhere function that I’m running in pmap, which works well except for the occasional glitch. I have a theory that said glitch is related to an I/O issue when multiple processes try to read/write files at the same time. (The exact nature of the glitch is not important at this juncture).

To try to isolate this I want to mutex the read/write functions to ensure that only one process can do I/O stuff at any given time. Is this possible with pmap? I tried creating a global SpinLock, but of course the individual processes don’t have access to it.

What’s the appropriate way to do this?

2

One way is to use lock files/dirs to synchronize processes, e.g. something like:

function acquire_lock(path)
    try
        mkdir(path)   # will only succeed if lock isn't acquired by someone else
        return true
    catch
        return false 
    end
end

function release_lock(path)
    try
        rm(path)
    catch
    end
end

function my_process()
    try
        while !acquire_lock(...)
            sleep(0.1)
        end
        # do the stuff
    finally
        release_lock(...)
    end
end

Of course, instead of files you can use another coordinating process, external service like Redis or whatever. Just make sure that lock acquisition is atomic and the process is guaranteed to releases it once the job is done.

3

That’ll do the job, of course. I was hoping for something built-in, but you can’t have everything…

Thanks!