Understanding message passing with pmap

I think (?) the default interprocess serialization/deserialization occurs via sockets and so it will be much faster than writing to disk. I would recommend sticking to standard Julia tooling here and rely on its (implicit) serialization for you (don’t prematurely optimize until you’re really sure it’s a big problem). If it turns out the overhead from the interprocess transfers is larger than the time saved from distributing to multiple workers, then consider whether a multiprocess model is the right fit here.

Also: be careful with the terminology. You said ‘threads’ in your recent comment but Distributed module is about processes. Julia also has threads, and these all share the same memory space, so data written in one thread can be visible in other threads (but beware race conditions, etc.). Threading is limited to within one node, however, and so won’t scale to multi-node architectures.