Julia has threads, but processes are better, providing isolation, and apparently help different languages work together (though Julia can interop with e.g. Python, such in-process is a risk, of overwriting not just your own memory, but for the other language).
This seems really cool, and Julia should support (or some Julia package):
Thread-speed, lock-free shared memory IPC in C11 .. Bindings share one wire format, so processes in different languages can exchange data over the same channel. The first-party bindings are mizu for R and pymizu for Python. The Writing a binding section covers other languages.
libmizu needs a C11 compiler and a 64-bit platform: the wire formats depend on lock-free 64-bit atomics. On Linux, kernel 5.3 or later is required (for pidfd_open).
From libmizu/DESIGN.md at main · shikokuchuo/libmizu · GitHub
Each process holds an exclusive lock on a file for its full lifetime: an flock on POSIX, LockFileEx on Windows. The kernel releases the lock on each exit path. The lock is fd-scoped, so PID reuse cannot fake “alive” .. Linux requires kernel >= 5.3 (pidfd_open, no fallback)
I didn’t think POSIX could lock files, only Windows, so I’m not sure how this is done, with pidfd_open or just lock files?
pidfd_open requires 5.10 or more for some things, but here only 5.3:
the POSIX locking API is the most portable one and in theory works across NFS. It can do byte-range locking. So much on the good side. On the bad side there’s a lot more however: locks are bound to processes, not file descriptors. That means that this logic cannot be used in threaded environments unless combined with a process-local mutex. This is hard to get right, especially in libraries that do not know the environment they are run in, i.e. whether they are used in threaded environments or not. The worst part however is that POSIX locks are automatically released if a process calls close() on any (!) of its open file descriptors for that file. That means that when one part of a program locks a file and another by coincidence accesses it too for a short time, the first part’s lock will be broken and it won’t be notified about that.
..
- The other API for POSIX file locks: lockf() is another API for the same mechanism and suffers by the same problems. One wonders why there are two APIs for the same messed up interface.
- BSD locking based on flock(). The semantics of this kind of locking are much nicer than for POSIX locking: locks are bound to file descriptors, not processes.
I suppose this is all good for that project; “flock” is advisory, but made to work.