I found the cause of the problem (and a partial solution). Looking at the strace (yes it was buffered) gave me a hunch to check the type of stdin/stdout when redirecting. Its type is indeed different (IOStream) from interactive use (Base.TTY). This in turn made me check for type (in)stability, and that seems to be the culprit. Using
f = stdin::IOStream
o = stdout::IOStream
solves the problem when redirecting. However, it gives an error when piping, stdout having yet another type in this case (Base.PipeEndpoint)
Allowing the function to specialize by giving stdin and stdout as parameters also seems to work (same speed as files):
function csv2tsv(f,o)
...
end
csv2tsv(stdin,stdout)
and without the error when used in a pipe.
However, although use in a pipe works, it becomes unbearably slow (~60 times slower than redirect to file): strace shows that the Base.PipeEndpoint stdout type does not buffer, and sprinkles 2 epoll_wait calls between each write (of one character)!
Does anybody know why this type shifting of stdin/stdout is happening? Redirecting and piping is quite essential (and extremely useful) on Unix systems, and this shifting type seems to be interfering badly with it.
It seems to me the standard streams should always simply be streams, regardless of the connection.