Hi, I’m having an issue with running an external command from Julia. I’m trying to read XML from a process’s output stream and parse it with LibExpat. The problem I’m having is that Julia freezes indefinitely at the first run. When I Ctrl+C (staying in the same REPL instance) and run again, it works as expected. I’m able to reproduce the problem with the following code snippet:
open(`cat about-500kb-of-xml.osm`, "r") do io
println(readline(io)) # should just print the first line of the xml file, but freezes at the first run, works fine the second time
end
When it’s frozen, I can see the cat process as a subprocess of Julia. The file size seems to matter; I don’t experience any problems with smaller files. The stacktrace when Ctrl+C’ing is:
[1] stream_wait(::Base.PipeEndpoint, ::Condition, ::Vararg{Condition,N} where N) at ./stream.jl:42
[2] close(::Base.PipeEndpoint) at ./stream.jl:326
[3] open(::Cmd, ::String, ::Base.DevNullStream) at ./process.jl:579
[4] open(::Cmd, ::String) at ./process.jl:575
[5] open(::##3#4, ::Cmd, ::String) at ./process.jl:599
Any idea what might be causing the freeze? Julia version 0.6.2. Thanks
Is cat just a stand in for a more complex command here? If it’s not, then you should just open the file itself for reading, as in
open("about-500kb-of-xml.osm", "r") do io
println(readline(io))
end
Julia output is not line buffered, so if you want to flush the output after each line, you’ll need an explicit call to flush(STDOUT). It’s possible that cat’s (or whatever command’s) output buffering is conflicting with Julia’s input buffering somehow, but it’s hard to say. Try it without the unnecessary cat process and/or with the flush call and see which if either helps.
cat is indeed a stand-in for a more complex command. The command itself converts binary-encoded OpenStreetMap data into XML. I used cat to provide an easily (assumably) reproducible example.
Everything works as expected when directly reading from a file.
I’d like to avoid creating the file explicitly, though, as the XML files easily grow a few 100MBs in size.
With respect to the buffering, the problem occurs regardless of what I do in the do-expression. I used a print statement for simplicity. The freeze seems to occur even before the do-expression is compiled (JIT magic).
