# Better support for running external commands
**URL:** https://discourse.julialang.org/t/better-support-for-running-external-commands/44933
**Category:** New to Julia
**Tags:** run, cmd
**Created:** [August 14, 2020, 2:18pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933 "2020-08-14T14:18:46Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)
#### Post date: [August 15, 2020, 11:09pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/21 "2020-08-15T23:09:15Z")
> [@Amin\_Yahyaabadi](#):
>
> ```julia
> # this checks if Manifest is tracked by Git, and if so it reverts it to the final state
> git ls-files 'Manifest.toml' | grep . && git checkout -- 'Manifest.toml'
>
> ```
What about something like this (untested, might need tweaking)?
```julia
if !isempty(readchomp(`git ls-files Manifest.toml`))
run(`git checkout -- Manifest.toml`)
end
```
For examples of doing git stuff from Julia, see e.g. [GitHub - JuliaRegistries/RegistryTools.jl: Functionality for modifying Julia package registry files](https://github.com/JuliaRegistries/RegistryTools.jl) and [GitHub - GunnarFarneback/LocalRegistry.jl: Create and maintain local registries for Julia packages.](https://github.com/GunnarFarneback/LocalRegistry.jl) (both `src` and `test` directories).
If you want to do the control flow with shell/cmd syntax, you indeed need to run it with e.g. `bash` or `cmd`. I prefer just calling the `git` external command and handle the control flow with Julia code.
---
### Author:  [@Amin\_Yahyaabadi](https://discourse.julialang.org/u/Amin_Yahyaabadi)
#### Post date: [August 15, 2020, 11:36pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/22 "2020-08-15T23:36:20Z")
> [@GunnarFarneback](#):
>
> If you want to do the control flow with shell/cmd syntax, you indeed need to run it with e.g. `bash` or `cmd` . I prefer just calling the `git` external command and handle the control flow with Julia code.
Thanks for the suggestion. For me personally this is fine and I prefer using Julia.
However, the main goal and topic of this thread were to mention the issues and lack of stability for running external commands. The idea of `run` is nice but saying that [shelling out sucks](https://julialang.org/blog/2012/03/shelling-out-sucks/) is just a unilateral opinion. Hopefully, this reveals some of the issues that `run` and `Cmd` have and helps to fix them, or even better we see a direct interface for calling external commands (which does not require rewriting the code in Julia, applying workarounds or modifying the commands).
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [August 16, 2020, 5:28am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/23 "2020-08-16T05:28:02Z")
> [@Amin\_Yahyaabadi](#):
>
> main goal and topic of this thread were to mention the issues and lack of stability for running external commands
I am confused, I thought that you were essentially missing shell syntax in `Cmd`. How does that qualify as “lack of stability”? AFAIK it has never been supported.
---
### Author:  [@Amin\_Yahyaabadi](https://discourse.julialang.org/u/Amin_Yahyaabadi)
#### Post date: [August 16, 2020, 7:17am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/24 "2020-08-16T07:17:03Z")
Yes. Maybe lack of a “feature” is a better word. Part of it is incomplete documentation. For example, Windows support is not considered in the documentation altogether.
---
### Author:  [@StevenSiew](https://discourse.julialang.org/u/StevenSiew)
#### Post date: [August 16, 2020, 9:22am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/25 "2020-08-16T09:22:35Z")
This works perfectly
```julia
julia> bashit(str) = run(`bash -c "$str"`)
bashit (generic function with 1 method)
julia> bashit("julia -e \"using Printf;@printf(\\\"%f\\n\\\",1)\"")
1.000000
Process(`bash -c 'julia -e "using Printf;@printf(\"%f\n\",1)"'`, ProcessExited(0))
julia> bashit(raw"""julia -e "using Printf;@printf(\\"%f\\n\\",1)" """)
1.000000
Process(`bash -c 'julia -e "using Printf;@printf(\"%f\n\",1)" '`, ProcessExited(0))
```
---
### Author:  [@StevenSiew](https://discourse.julialang.org/u/StevenSiew)
#### Post date: [August 16, 2020, 9:57am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/26 "2020-08-16T09:57:14Z")
If you really really need to execute some shell commands then
1. write to a textfile like “qqq.sh”
then execute it. This way, you are just running a shell script.
```julia
function execute(cmd::Cmd)
out = Pipe()
err = Pipe()
process = run(pipeline(ignorestatus(cmd), stdout = out, stderr = err))
close(out.in)
close(err.in)
stdout = @async String(read(out))
stderr = @async String(read(err))
wait(process)
return (
stdout = fetch(stdout),
stderr = fetch(stderr),
code = process.exitcode,
)
end
execute(`bash -c "cd /Users/ssiew/juliascript/coronavirus; pwd; ./qqq.sh"`)
```
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [August 16, 2020, 1:09pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/27 "2020-08-16T13:09:38Z")
> [@Amin\_Yahyaabadi](#):
>
> The idea of `run` is nice but saying that [shelling out sucks](https://julialang.org/blog/2012/03/shelling-out-sucks/) is just a unilateral opinion.
As far as I can see, everything in this thread has supported the argument that “shelling out sucks”. As demonstrated above, it is non-portable, fragile to metacharacters, and slow. All of the problems in the [example above](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/11) would have been avoided by simply using `run(`julia -e $ex`)`.
When you are using Julia, you have a programming language that is perfectly capable of launching processes, creating pipelines, etcetera. Why should it default to launching processes by resorting to a _second_ programming language (the shell), especially one with all of these deficiencies? And if you do choose to spawn a shell, why blame Julia for the deficiencies of the _shell’s_ programming language?
The main argument in favor of shelling out, as far as I can tell, is that people coming from other programming languages are used to thinking of the shell as the only convenient way to control other processes. It’s a variant of the [“why can’t Julia use the syntax of my old language” FAQ](https://docs.julialang.org/en/v1/manual/faq/#Why-don't-you-compile-Matlab/Python/R/%E2%80%A6-code-to-Julia?-1) — yes, to get the benefits of a new language, sometimes you need to learn new syntax. There are probably many opportunities in Julia to introduce nicer syntax for some shell-like operations, e.g. binary operators for pipelines, but simply adopting `bash` syntax wholesale is not an option — Julia can’t become `bash` without inheriting the limitations of `bash`.
---
### Author:  [@Amin\_Yahyaabadi](https://discourse.julialang.org/u/Amin_Yahyaabadi)
#### Post date: [August 16, 2020, 11:28pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/28 "2020-08-16T23:28:30Z")
Thanks. I see you have escaped a lot of things! Does calling `raw` two times help with this? The reason I am saying this is because the Julia script is unknown in my problem.
---
### Author:  [@Amin\_Yahyaabadi](https://discourse.julialang.org/u/Amin_Yahyaabadi)
#### Post date: [August 16, 2020, 11:30pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/29 "2020-08-16T23:30:04Z")
Yes, that’s the most robust solution. Thanks!
---
### Author:  [@Amin\_Yahyaabadi](https://discourse.julialang.org/u/Amin_Yahyaabadi)
#### Post date: [August 16, 2020, 11:33pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/30 "2020-08-16T23:33:26Z")
> [@stevengj](#):
>
> As far as I can see, everything in this thread has supported the argument that “shelling out sucks”. As demonstrated above, it is non-portable, fragile to metacharacters, and slow. All of the problems in the [example above](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/11) would have been avoided by simply using `run(`julia -e $ex`)` .
I am not sure how this has supported those arguments, but if this is how it seems I am fine.
If you look at the source code of BinaryBuilder.jl, you see that they shell out! If Julia is perfect in calling the external programs, I don’t see why BinaryBuilder should do this. Why don’t they make everyone rewrite the build scripts in the new Julia compatible syntax?
My answer is that you can’t convince every programmer to switch to the `run` compatible syntax, and to build Yggdrasil programs you should use a common shell language (like bash). If you wanted to rewrite all of these **just in `run`** , we did not have so many recipes in Yggdrasil. Imagine rewriting [this](https://github.com/JuliaPackaging/Yggdrasil/blob/fcdc0be9675190270e65e20285492f5bc72f832c/A/armadillo/build_tarballs.jl#L14-L65) in separate `run` calls!
>
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [August 17, 2020, 6:16am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/31 "2020-08-17T06:16:01Z")
> [@Amin\_Yahyaabadi](#):
>
> If you look at the source code of BinaryBuilder.jl, you see that they shell out!
I’d like to point out that the commands you’re referring to are executed within the build environment which is a very controlled Linux environment where we’re sure that bash is available. It’s impossible to write such commands on an arbitrary system where you don’t what shell is there and what syntax it uses.
Once again, you’re showing examples that actually disprove your point.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [August 17, 2020, 6:54am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/32 "2020-08-17T06:54:05Z")
> [@Amin\_Yahyaabadi](#):
>
> Imagine rewriting [this](https://github.com/JuliaPackaging/Yggdrasil/blob/fcdc0be9675190270e65e20285492f5bc72f832c/A/armadillo/build_tarballs.jl#L14-L65) in separate `run` calls!
The snippet you link mostly builds variables (`FLAGS`, `SYMB_DEFS`, …), then calls out to `cmake` and `make`.
Idiomatic Julia code could just build up these variables in Julia. I am not saying that this should be done (don’t change what works, etc), just pointing out that it would not be nearly as onerous as you seem to imagine.
---
### Author:  [@StevenSiew](https://discourse.julialang.org/u/StevenSiew)
#### Post date: [August 17, 2020, 9:35am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/33 "2020-08-17T09:35:01Z")
Here is another way of doing it
```julia
function escstr(str)
arr = collect(str); # Split string into an array of characters
resultstr = Array{eltype(arr)}(undef,0)
# Foreach character do any substitutions neccesary
for c in arr
if c == '"'
push!(resultstr,'\\','"')
elseif c == '\\'
push!(resultstr,'\\','\\')
else
push!(resultstr,c)
end
end
return join(resultstr)
end
bashit(str) = run(`bash -c "$str"`)
julia> dq="\"";
julia> jcmd = escstr(raw"""using Printf; @printf("%f\n",1.0)""");
julia> bashit("julia -e $(dq)$(jcmd)$(dq)")
1.000000
Process(`bash -c 'julia -e "using Printf; @printf(\"%f\\n\",1.0)"'`, ProcessExited(0))
```
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [August 17, 2020, 10:21am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/34 "2020-08-17T10:21:20Z")
> [@Amin\_Yahyaabadi](#):
>
> Imagine rewriting [this](https://github.com/JuliaPackaging/Yggdrasil/blob/fcdc0be9675190270e65e20285492f5bc72f832c/A/armadillo/build_tarballs.jl#L14-L65) in separate `run` calls!
I don’t see why that would be particularly hard? The code would probably look nicer and be easier to maintain. Just like most things written in Julia are 🙂
But note that what you linked there is literally a recording of a bash session. That recording is supposed to be replayed in a small controlled environment with the same version of everything as when it was recorded.
Having build recipes written in `bash` instead of Julia simplifies the environment where they can be executed (Julia can be a bit of a heavy dependency) and also makes it easier for other languages to use the recipes. It doesn’t mean anything for what `run` in Julia itself should do though.
---
### Author:  [@Amin\_Yahyaabadi](https://discourse.julialang.org/u/Amin_Yahyaabadi)
#### Post date: [August 19, 2020, 5:34pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/35 "2020-08-19T17:34:03Z")
> [@kristoffer.carlsson](#):
>
> Having build recipes written in `bash` instead of Julia simplifies the environment where they can be executed (Julia can be a bit of a heavy dependency) and also makes it easier for other languages to use the recipes.
I have the same rationale behind using bash in the code. If the code is not written and it is being written for the first time, it is easier to write in Julia. However, that is not usually the case.
---
### Author:  [@Amin\_Yahyaabadi](https://discourse.julialang.org/u/Amin_Yahyaabadi)
#### Post date: [August 19, 2020, 5:51pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/36 "2020-08-19T17:51:10Z")
Thanks!
I am thinking of a `Execute.jl` to collect all these workarounds and also trying to write a minimal direct interface by using `jl_spawn`.
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [August 19, 2020, 6:50pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/37 "2020-08-19T18:50:45Z")
> [@Amin\_Yahyaabadi](#):
>
> I have the same rationale behind using bash in the code.
So call bash then? Like, what’s the problem?
---
### Author:  [@RogerP](https://discourse.julialang.org/u/RogerP)
#### Post date: [October 24, 2021, 4:03pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/38 "2021-10-24T16:03:52Z")
Maybe I want (and this is not hypothetical) to query the status of some process Julia is interacting with. I am using the [QuestDB](https://questdb.io/) database. I need to know if it’s live or my code will error, I would also like to know its PID so that I can kill the process. The bash command is `ps -ef | grep questdb | awk '{print $2}'`, if I use `raw` like this
```julia
julia> raw"""ps -ef | grep questdb | awk '{print $2}'"""
```
I get an escaped `$2` :
```julia
"ps -ef | grep questdb | awk '{print \$2}'"
```
I think it’s reasonable to want to do this without Julia going to the trouble of escaping my $
---
### Author:  [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)
#### Post date: [October 24, 2021, 4:25pm UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/39 "2021-10-24T16:25:16Z")
> [@RogerP](#):
>
> I think it’s reasonable to want to do this without Julia going to the trouble of escaping my $
It’s not escaped. That’s just how strings are printed.
```julia
julia> s = raw"""echo "foo bar" | awk '{print $2}'"""
"echo \"foo bar\" | awk '{print \$2}'"
julia> print(s)
echo "foo bar" | awk '{print $2}'
julia> run(`sh -c $s`)
bar
Process(`sh -c "echo \"foo bar\" | awk '{print \$2}'"`, ProcessExited(0))
```
---
### Author:  [@RogerP](https://discourse.julialang.org/u/RogerP)
#### Post date: [October 25, 2021, 10:43am UTC](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933/41 "2021-10-25T10:43:28Z")
My apologies – it works just fine. I gave up to soon. Thank you.
[Previous page](https://discourse.julialang.org/t/better-support-for-running-external-commands/44933.md?page=1)