Well, using a pipeline with dd and a defined small blocksize seems to hide these issues:
open(load, pipeline(`cat libLAS_1.2.laz`,`dd bs=8`), "r")
Works for me over a thousand times, although @visr said he experienced one hang on bs=4.
The larger the blocksize, the more often I experience deadlocks. A blocksize of 64 instantly hangs for me.
The fact that it instantly hangs at 64kb is because of the maximum kernel pipe buffer size. It just can’t buffer more, and Julia seems to wait to receive more data (not taking the limit into account?). There is an old issue for this, but it’s closed now. https://github.com/JuliaLang/julia/issues/8789. Maybe this should just be documented.
I can’t say where things go wrong. On github we have:
Issue of writing to pipe on Mac: https://github.com/JuliaLang/julia/issues/20812
Issue of readandwrite deadlock: https://github.com/JuliaLang/julia/issues/18840
Issue of intermittent failing deadlock in tests: https://github.com/JuliaLang/julia/issues/14747
@Keno Could you shed some light on this?
At the moment it seems that pipes (and thus interaction with external programs) can’t be used reliably for anything that reads/writes more than a few kb.