File IO Buffers too small?

[kristoffer.carlsson]

@mwlidar

My current solution is to allocate a large buffer and fill it with a separate thread. But the usage is quite cumbersome with all the extra synchronization handling and the code has to be specially adapted.

You can use GitHub - JuliaIO/BufferedStreams.jl: Fast composable IO streams.

But indeed, I have also found the performance of IOStream to be unacceptable for many IO tasks where you typically read many small values. One issue with IOStream is that it takes a lock on every IO operation in order to be thread-safe.

For binary files, I typically resort to memory mapping the file and wrapping it in an IOBuffer.

So instead of

io = open("some very large file.dat")

I do

using Mmap
io = IOBuffer(Mmap.mmap("some very large file.dat"))