That was definitely one thing slowing the sequential write!
Turning off the lock gave a stable 5x speed-up - making it a reasonable solution if one has no other choice.
Still, bulk write is much faster for files below 20-50 MB.
When I glimpsed into the Base IO code, it seems Julia specializes writing byte arrays directly into a function called unsafe_write()
, as opposed to normal write()
in other cases. Both of them make a call to some C function, so this is as far as I can go.
Interestingly, writing in batches seems most efficient when there is a small number of them (10 worked for me the best), as long as single batch does not exceed the 20-50 MB limit. After that it gets hit by the same penalty as bulk write.
So for bigger files, when you want to write as fast as possible, 10MB batches where the fastest in my tests.
Below is the graph comparing, sequential (with and without lock), bulk, 10k batch, and batch = 1/10 of the file.