# Collecting all output from shell commands

**URL:** <https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592>\
**Category:** General Usage\
**Tags:** shell\
**Created:** [September 28, 2018, 7:35am UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592 "2018-09-28T07:35:43Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![peteristhegreat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/peteristhegreat/32/8220_2.png) [@peteristhegreat](https://discourse.julialang.org/u/peteristhegreat)\
**Post date:** [August 2, 2023, 3:42pm UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592/21 "2023-08-02T15:42:19Z")

</div>

I added support for pipelines going into communicate and collecting the max of the error codes. Thanks for posting the original function.

Also I referenced it over here: [powershell - How do I prevent an error message when using get-process for an process that does not run? - Stack Overflow](https://stackoverflow.com/questions/58948169/how-do-i-prevent-an-error-message-when-using-get-process-for-an-process-that-doe)

```julia
function communicate(cmd::Base.AbstractCmd, input="")
    inp = Pipe()
    out = Pipe()
    err = Pipe()

    process = run(pipeline(cmd, stdin=inp, stdout=out, stderr=err), wait=false)
    close(out.in)
    close(err.in)

    stdout = @async String(read(out))
    stderr = @async String(read(err))
    write(process, input)
    close(inp)
    wait(process)
    if process isa Base.ProcessChain
        exitcode = maximum([p.exitcode for p in process.processes])
    else
        exitcode = process.exitcode
    end

    return (
        stdout = fetch(stdout),
        stderr = fetch(stderr),
        code = exitcode
    )
end

```

---

<div class="post-metadata">

**Author:** ![phantom](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@phantom](https://discourse.julialang.org/u/phantom)\
**Post date:** [March 19, 2024, 9:16am UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592/22 "2024-03-19T09:16:02Z")

</div>

Would `@async` in `stdout` and `stderr` now be replaced with `Threads.@spawn` given the [warning](https://docs.julialang.org/en/v1/base/parallel/#Base.@async) in the documentation, or would that not be an issue/appropriate here? Just asking given that this function is still likely to be widely referenced.

---

<div class="post-metadata">

**Author:** ![jameson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jameson/32/23_2.png) [@jameson](https://discourse.julialang.org/u/jameson)\
**Post date:** [April 2, 2024, 12:33pm UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592/23 "2024-04-02T12:33:17Z")

</div>

Yes, though now you might want to use an IOBuffer there instead of Pipe, which will handle doing that internally

---

<div class="post-metadata">

**Author:** ![phantom](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@phantom](https://discourse.julialang.org/u/phantom)\
**Post date:** [April 3, 2024, 4:47am UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592/24 "2024-04-03T04:47:31Z")

</div>

Sorry just to clarify did you mean that with IOBuffers I don’t have to worry about using either `@async` or `@spawn`? something like

```julia
function communicate(cmd::Cmd)
    out = IOBuffer()
    err = IOBuffer()

    process = run(pipeline(cmd, stdout=out, stderr=err), wait=false)
    wait(process)

    seekstart(out)
    seekstart(err)
    
    if process isa Base.ProcessChain
        exitcode = maximum([p.exitcode for p in process.processes])
    else
        exitcode = process.exitcode
    end

    return (
        stdout = String(take!(out)),
        stderr = String(take!(err)),
        code = exitcode
    )
end

```

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [April 3, 2024, 10:03am UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592/25 "2024-04-03T10:03:20Z")

</div>

Only starting on Julia v1.11 right?

---

<div class="post-metadata">

**Author:** ![jameson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jameson/32/23_2.png) [@jameson](https://discourse.julialang.org/u/jameson)\
**Post date:** [April 3, 2024, 12:29pm UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592/26 "2024-04-03T12:29:17Z")

</div>

That is both right. And both wait and pipeline are redundant there, since those are also options directly of run, so can be simplified further. And the seekstart has no effect either.

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [September 23, 2024, 2:51pm UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592/27 "2024-09-23T14:51:32Z")

</div>

I just wanted to add this version as it can be useful for someone who also want to process the shell output in a streaming fashion. I wanted to had a version that doesn’t just do: `String(take!(out))` and return everything at once.

In the example, it is just `println` for processing it.

```julia
using Base.Threads

function run_command_async(cmd_str)
    cmd = `bash -c $cmd_str`
    
    # Create a pipe to capture the output
    out_pipe = Pipe()
    err_pipe = Pipe()

    # Run the command asynchronously
    process = run(pipeline(cmd, stdout=out_pipe, stderr=err_pipe), wait=false)
    close(out_pipe.in)
    close(err_pipe.in)
		
    # Stream output in real-time
    @async while !eof(out_pipe)
        line = readline(out_pipe)
        println(line)
    end
    @async while !eof(err_pipe)
        line = readline(err_pipe)
        println(line)
    end

    # Wait for the command to finish
    wait(process)

    # Print any remaining output
    for line in eachline(out_pipe)
        println(line)
    end
end

cmd_str = "for i in {1..3}; do echo \$i; sleep 0.5; done"
run_command_async(cmd_str)

println("\nCommand execution completed.")

```

Ofc lot of customization can be done here…

---

<div class="post-metadata">

**Author:** ![jameson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jameson/32/23_2.png) [@jameson](https://discourse.julialang.org/u/jameson)\
**Post date:** [September 28, 2024, 7:34pm UTC](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592/28 "2024-09-28T19:34:32Z")

</div>

That is useful, but note that you used out\_pipe in two places which causes a data race there. You can give the async blocks a name and wait on those, instead of copying the code, to fix that race.

[Previous page](https://discourse.julialang.org/t/collecting-all-output-from-shell-commands/15592.md?page=1)