Collecting all output from shell commands

General Usage
21

I added support for pipelines going into communicate and collecting the max of the error codes. Thanks for posting the original function.

Also I referenced it over here: powershell - How do I prevent an error message when using get-process for an process that does not run? - Stack Overflow

function communicate(cmd::Base.AbstractCmd, input="")
    inp = Pipe()
    out = Pipe()
    err = Pipe()

    process = run(pipeline(cmd, stdin=inp, stdout=out, stderr=err), wait=false)
    close(out.in)
    close(err.in)

    stdout = @async String(read(out))
    stderr = @async String(read(err))
    write(process, input)
    close(inp)
    wait(process)
    if process isa Base.ProcessChain
        exitcode = maximum([p.exitcode for p in process.processes])
    else
        exitcode = process.exitcode
    end

    return (
        stdout = fetch(stdout),
        stderr = fetch(stderr),
        code = exitcode
    )
end
1 Like
22

Would @async in stdout and stderr now be replaced with Threads.@spawn given the warning in the documentation, or would that not be an issue/appropriate here? Just asking given that this function is still likely to be widely referenced.

2 Likes
23

Yes, though now you might want to use an IOBuffer there instead of Pipe, which will handle doing that internally

1 Like
24

Sorry just to clarify did you mean that with IOBuffers I donā€™t have to worry about using either @async or @spawn? something like

function communicate(cmd::Cmd)
    out = IOBuffer()
    err = IOBuffer()

    process = run(pipeline(cmd, stdout=out, stderr=err), wait=false)
    wait(process)

    seekstart(out)
    seekstart(err)
    
    if process isa Base.ProcessChain
        exitcode = maximum([p.exitcode for p in process.processes])
    else
        exitcode = process.exitcode
    end

    return (
        stdout = String(take!(out)),
        stderr = String(take!(err)),
        code = exitcode
    )
end
25

Only starting on Julia v1.11 right?

26

That is both right. And both wait and pipeline are redundant there, since those are also options directly of run, so can be simplified further. And the seekstart has no effect either.

1 Like
27

I just wanted to add this version as it can be useful for someone who also want to process the shell output in a streaming fashion. I wanted to had a version that doesnā€™t just do: String(take!(out)) and return everything at once.

In the example, it is just println for processing it.

using Base.Threads

function run_command_async(cmd_str)
    cmd = `bash -c $cmd_str`
    
    # Create a pipe to capture the output
    out_pipe = Pipe()
    err_pipe = Pipe()

    # Run the command asynchronously
    process = run(pipeline(cmd, stdout=out_pipe, stderr=err_pipe), wait=false)
    close(out_pipe.in)
    close(err_pipe.in)
		
    # Stream output in real-time
    @async while !eof(out_pipe)
        line = readline(out_pipe)
        println(line)
    end
    @async while !eof(err_pipe)
        line = readline(err_pipe)
        println(line)
    end

    # Wait for the command to finish
    wait(process)

    # Print any remaining output
    for line in eachline(out_pipe)
        println(line)
    end
end

cmd_str = "for i in {1..3}; do echo \$i; sleep 0.5; done"
run_command_async(cmd_str)

println("\nCommand execution completed.")

Ofc lot of customization can be done hereā€¦

28

That is useful, but note that you used out_pipe in two places which causes a data race there. You can give the async blocks a name and wait on those, instead of copying the code, to fix that race.

1 Like