Better support for running external commands

New to Julia
,
21

What about something like this (untested, might need tweaking)?

if !isempty(readchomp(`git ls-files Manifest.toml`))
    run(`git checkout -- Manifest.toml`)
end

For examples of doing git stuff from Julia, see e.g. GitHub - JuliaRegistries/RegistryTools.jl: Functionality for modifying Julia package registry files and GitHub - GunnarFarneback/LocalRegistry.jl: Create and maintain local registries for Julia packages. (both src and test directories).

If you want to do the control flow with shell/cmd syntax, you indeed need to run it with e.g. bash or cmd. I prefer just calling the git external command and handle the control flow with Julia code.

4 Likes
22

Thanks for the suggestion. For me personally this is fine and I prefer using Julia.

However, the main goal and topic of this thread were to mention the issues and lack of stability for running external commands. The idea of run is nice but saying that shelling out sucks is just a unilateral opinion. Hopefully, this reveals some of the issues that run and Cmd have and helps to fix them, or even better we see a direct interface for calling external commands (which does not require rewriting the code in Julia, applying workarounds or modifying the commands).

23

I am confused, I thought that you were essentially missing shell syntax in Cmd. How does that qualify as “lack of stability”? AFAIK it has never been supported.

24

Yes. Maybe lack of a “feature” is a better word. Part of it is incomplete documentation. For example, Windows support is not considered in the documentation altogether.

25

This works perfectly

julia> bashit(str) = run(`bash -c "$str"`)
bashit (generic function with 1 method)

julia> bashit("julia -e \"using Printf;@printf(\\\"%f\\n\\\",1)\"")
1.000000
Process(`bash -c 'julia -e "using Printf;@printf(\"%f\n\",1)"'`, ProcessExited(0))

julia> bashit(raw"""julia -e "using Printf;@printf(\\"%f\\n\\",1)" """)
1.000000
Process(`bash -c 'julia -e "using Printf;@printf(\"%f\n\",1)" '`, ProcessExited(0))
6 Likes
26

If you really really need to execute some shell commands then

  1. write to a textfile like “qqq.sh”

then execute it. This way, you are just running a shell script.

function execute(cmd::Cmd)
    out = Pipe()
    err = Pipe()

    process = run(pipeline(ignorestatus(cmd), stdout = out, stderr = err))
    close(out.in)
    close(err.in)

    stdout = @async String(read(out))
    stderr = @async String(read(err))
    wait(process)
    return (
        stdout = fetch(stdout),
        stderr = fetch(stderr),
        code = process.exitcode,
    )
end

execute(`bash -c "cd /Users/ssiew/juliascript/coronavirus; pwd; ./qqq.sh"`)
6 Likes
27

As far as I can see, everything in this thread has supported the argument that “shelling out sucks”. As demonstrated above, it is non-portable, fragile to metacharacters, and slow. All of the problems in the example above would have been avoided by simply using run(`julia -e $ex`).

When you are using Julia, you have a programming language that is perfectly capable of launching processes, creating pipelines, etcetera. Why should it default to launching processes by resorting to a second programming language (the shell), especially one with all of these deficiencies? And if you do choose to spawn a shell, why blame Julia for the deficiencies of the shell’s programming language?

The main argument in favor of shelling out, as far as I can tell, is that people coming from other programming languages are used to thinking of the shell as the only convenient way to control other processes. It’s a variant of the “why can’t Julia use the syntax of my old language” FAQ — yes, to get the benefits of a new language, sometimes you need to learn new syntax. There are probably many opportunities in Julia to introduce nicer syntax for some shell-like operations, e.g. binary operators for pipelines, but simply adopting bash syntax wholesale is not an option — Julia can’t become bash without inheriting the limitations of bash.

17 Likes
28

Thanks. I see you have escaped a lot of things! Does calling raw two times help with this? The reason I am saying this is because the Julia script is unknown in my problem.

29

Yes, that’s the most robust solution. Thanks!

30

I am not sure how this has supported those arguments, but if this is how it seems I am fine.

If you look at the source code of BinaryBuilder.jl, you see that they shell out! If Julia is perfect in calling the external programs, I don’t see why BinaryBuilder should do this. Why don’t they make everyone rewrite the build scripts in the new Julia compatible syntax?

My answer is that you can’t convince every programmer to switch to the run compatible syntax, and to build Yggdrasil programs you should use a common shell language (like bash). If you wanted to rewrite all of these just in run, we did not have so many recipes in Yggdrasil. Imagine rewriting this in separate run calls!

31

I’d like to point out that the commands you’re referring to are executed within the build environment which is a very controlled Linux environment where we’re sure that bash is available. It’s impossible to write such commands on an arbitrary system where you don’t what shell is there and what syntax it uses.

Once again, you’re showing examples that actually disprove your point.

5 Likes
32

The snippet you link mostly builds variables (FLAGS, SYMB_DEFS, …), then calls out to cmake and make.

Idiomatic Julia code could just build up these variables in Julia. I am not saying that this should be done (don’t change what works, etc), just pointing out that it would not be nearly as onerous as you seem to imagine.

1 Like
33

Here is another way of doing it

    function escstr(str)
        arr = collect(str); # Split string into an array of characters
        resultstr = Array{eltype(arr)}(undef,0)
        # Foreach character do any substitutions neccesary        
        for c in arr
            if c == '"'
                push!(resultstr,'\\','"')
            elseif c == '\\'
                push!(resultstr,'\\','\\')
            else
                push!(resultstr,c)
            end
        end
        return join(resultstr)
    end


    bashit(str) = run(`bash -c "$str"`)

julia> dq="\"";

julia> jcmd = escstr(raw"""using Printf; @printf("%f\n",1.0)""");

julia> bashit("julia -e $(dq)$(jcmd)$(dq)")
1.000000
Process(`bash -c 'julia -e "using Printf; @printf(\"%f\\n\",1.0)"'`, ProcessExited(0))
3 Likes
34

I don’t see why that would be particularly hard? The code would probably look nicer and be easier to maintain. Just like most things written in Julia are :slight_smile:

But note that what you linked there is literally a recording of a bash session. That recording is supposed to be replayed in a small controlled environment with the same version of everything as when it was recorded.
Having build recipes written in bash instead of Julia simplifies the environment where they can be executed (Julia can be a bit of a heavy dependency) and also makes it easier for other languages to use the recipes. It doesn’t mean anything for what run in Julia itself should do though.

3 Likes
35

I have the same rationale behind using bash in the code. If the code is not written and it is being written for the first time, it is easier to write in Julia. However, that is not usually the case.

36

Thanks!

I am thinking of a Execute.jl to collect all these workarounds and also trying to write a minimal direct interface by using jl_spawn.

2 Likes
37

So call bash then? Like, what’s the problem?

1 Like
38

Maybe I want (and this is not hypothetical) to query the status of some process Julia is interacting with. I am using the QuestDB database. I need to know if it’s live or my code will error, I would also like to know its PID so that I can kill the process. The bash command is ps -ef | grep questdb | awk '{print $2}', if I use raw like this

julia> raw"""ps -ef | grep questdb | awk '{print $2}'"""

I get an escaped $2 :

"ps -ef | grep questdb | awk '{print \$2}'"

I think it’s reasonable to want to do this without Julia going to the trouble of escaping my $

39

It’s not escaped. That’s just how strings are printed.

julia> s = raw"""echo "foo bar" | awk '{print $2}'"""
"echo \"foo bar\" | awk '{print \$2}'"

julia> print(s)
echo "foo bar" | awk '{print $2}'
julia> run(`sh -c $s`)
bar
Process(`sh -c "echo \"foo bar\" | awk '{print \$2}'"`, ProcessExited(0))
2 Likes
41

My apologies – it works just fine. I gave up to soon. Thank you.