Better support for running external commands

[Amin_Yahyaabadi]

Currently, in Julia the run function is not as stable as the rest of the Julia.

You can’t run external commands with special characters. The different behavior of Cmd from String makes using it inconvenient. Windows support is not like other operating systems.

Running something as simple as the following in Julia requires a lot of workarounds.

# python
import os
os.system("echo hey/*")
os.system("echo hey && echo bye")
x="echo bye"
os.system(x)

Trying these in Julia:

1. Running Commands with special characters:

using Cmd direcrtly fails

julia> run(`echo hey/*`)
ERROR: LoadError: parsing command `echo hey/*`: special characters "#{}()[]<>|&*?~;" must be quoted in commands
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] shell_parse(::String, ::Bool; special::String) at .\shell.jl:100
 [3] @cmd(::LineNumberNode, ::Module, ::Any) at .\cmd.jl:389
in expression starting at REPL[3]:1

Using @cmd fails

julia> @cmd run"echo hey/*"
ERROR: LoadError: MethodError: no method matching shell_parse(::Expr; special="#{}()[]<>|&*?~;")
Closest candidates are:
  shell_parse(::AbstractString) at shell.jl:18 got unsupported keyword argument "special"
  shell_parse(::AbstractString, ::Bool; special) at shell.jl:18
Stacktrace:
 [1] @cmd(::LineNumberNode, ::Module, ::Any) at .\cmd.jl:389
in expression starting at REPL[6]:1

There is no @raw_cmd and @raw_string doesn’t work as stated in 3

julia> run(raw`echo hey/*`)
ERROR: LoadError: UndefVarError: @raw_cmd not defined
in expression starting at REPL[1]:1

julia> run(raw"echo hey/*")
ERROR: MethodError: no method matching run(::String)
Closest candidates are:
  run(::Base.AbstractCmd, ::Any...; wait) at process.jl:437
Stacktrace:
 [1] top-level scope at REPL[2]:1

julia> run(`$(raw"echo hey/*")`)
ERROR: IOError: could not spawn `'echo hey/*'`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] #585 at .\process.jl:112 [inlined]
 [3] setup_stdios(::Base.var"#585#586"{Cmd}, ::Array{Any,1}) at .\process.jl:196
 [4] _spawn at .\process.jl:111 [inlined]
 [5] run(::Cmd; wait::Bool) at .\process.jl:439
 [6] run(::Cmd) at .\process.jl:438
 [7] top-level scope at REPL[1]:1

In some cases, it is possible to explicitly escape the special characters but not always

# this works
julia> run(`echo hey/\*`)
hey/*
Process(`echo 'hey/*'`, ProcessExited(0))

# this doesn't
julia> run(`echo hi \&\& echo bye`)  # or  run(`echo hi '&&' echo bye`)
hi && echo bye
Process(`echo hi '&&' echo bye`, ProcessExited(0))

2. Out of the box Windows Support

This requires another hackery if you want to run this on Windows:

# happens in pwsh, powershell, cmd
julia> run(`echo hey/\*`)
ERROR: IOError: could not spawn `echo 'hey/*'`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] #585 at .\process.jl:112 [inlined]
 [3] setup_stdios(::Base.var"#585#586"{Cmd}, ::Array{Any,1}) at .\process.jl:196
 [4] _spawn at .\process.jl:111 [inlined]
 [5] run(::Cmd; wait::Bool) at .\process.jl:439
 [6] run(::Cmd) at .\process.jl:438
 [7] top-level scope at REPL[1]:1

julia> run(`cmd /c echo hey/\*`)
hey/*
Process(`cmd /c echo 'hey/*'`, ProcessExited(0))

3. Possibility of using String for building a command

julia> x = "echo bye"
"echo bye"

Using String directly fails

julia> Cmd(x)
ERROR: MethodError: no method matching Cmd(::String)
Closest candidates are:
  Cmd(::Array{String,1}) at cmd.jl:16
  Cmd(::Cmd; ignorestatus, env, dir, detach, windows_verbatim, windows_hide) at cmd.jl:21
  Cmd(::Cmd, ::Any, ::Any, ::Any, ::Any) at cmd.jl:18
Stacktrace:
 [1] top-level scope at REPL[40]:1

Trying interpolation

julia> run(`$x`)
ERROR: IOError: could not spawn `'echo bye'`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] #585 at .\process.jl:112 [inlined]
 [3] setup_stdios(::Base.var"#585#586"{Cmd}, ::Array{Any,1}) at .\process.jl:196
 [4] _spawn at .\process.jl:111 [inlined]
 [5] run(::Cmd; wait::Bool) at .\process.jl:439
 [6] run(::Cmd) at .\process.jl:438
 [7] top-level scope at REPL[31]:1

Trying @cmd

julia> @cmd($x)
ERROR: LoadError: MethodError: no method matching shell_parse(::Expr; special="#{}()[]<>|&*?~;")
Closest candidates are:
  shell_parse(::AbstractString) at shell.jl:18 got unsupported keyword argument "special"
  shell_parse(::AbstractString, ::Bool; special) at shell.jl:18
Stacktrace:
 [1] @cmd(::LineNumberNode, ::Module, ::Any) at .\cmd.jl:389
in expression starting at REPL[36]:1

julia> @cmd(x)
ERROR: LoadError: MethodError: no method matching shell_parse(::Symbol; special="#{}()[]<>|&*?~;")
Closest candidates are:
  shell_parse(::AbstractString) at shell.jl:18 got unsupported keyword argument "special"
  shell_parse(::AbstractString, ::Bool; special) at shell.jl:18
Stacktrace:
 [1] @cmd(::LineNumberNode, ::Module, ::Any) at .\cmd.jl:389
in expression starting at REPL[37]:1

julia> @cmd("$x")
ERROR: LoadError: MethodError: no method matching shell_parse(::Expr; special="#{}()[]<>|&*?~;")
Closest candidates are:
  shell_parse(::AbstractString) at shell.jl:18 got unsupported keyword argument "special"
  shell_parse(::AbstractString, ::Bool; special) at shell.jl:18
Stacktrace:
 [1] @cmd(::LineNumberNode, ::Module, ::Any) at .\cmd.jl:389
in expression starting at REPL[39]:1

My suggestion for 1 and 3: allow using String with run. This will fix 1 with the help of @raw_string. This will also fix 3 by directly sending the command to the system without putting ' around it.
My suggestion for 2: detect the host shell and adjust run accordignly. If not possible define a run2 which automatically finds pwsh, powershell, or cmd. In Julia 2, make run2 the default run.

[simeonschaub]

@Amin_Yahyaabadi

You can’t run external commands with special characters.

You can do this by quoting the command with single or double backticks.

@Amin_Yahyaabadi

using Cmd direcrtly fails

julia

This is because globs (e.g. * to match any string) don’t work in Julia (yet).

@Amin_Yahyaabadi

# this works
julia> run(`echo hey/\*`)
hey/*
Process(`echo 'hey/*'`, ProcessExited(0))

This only matches a path with the literal name *, which is probably not what you want.

@Amin_Yahyaabadi

# this doesn't
julia> run(`echo hi \&\& echo bye`)  # or  run(`echo hi '&&' echo bye`)
hi && echo bye
Process(`echo hi '&&' echo bye`, ProcessExited(0))

It does what you ask it to, it escapes && as an argument to echo. && for command chaining, like in bash, is also not currently supported.

@Amin_Yahyaabadi

3. Possibility of using String for building a command

julia> x = "echo bye"
"echo bye"

This is actually a feature, because it protects you from command injection. You usually don’t just want to run arbitrary strings as commands. If you really want to create a command from a string like this, you need to call Base.shell_parse directly, but this could definitely be documented better.

I do agree that there are still a lot of ways running external commands could be improved. To me, that would mean moving Glob.jl into base, supporting piping and chaining constructs like |, >, &&, etc. directly in command literals, and I’d also really like to see parse command interpolations at parse time · Issue #3150 · JuliaLang/julia · GitHub being addressed.

[giordano]

Note that all these features you’re talking about (globbing, piping, short circuiting) are shell’s features, julia doesn’t use the shell to parse the command, which is arguably a feature as it makes the code independent from the currently used shell

[simeonschaub]

Note however that this doesn’t mean, we can’t have those features in Julia eventually. Command literals are already their own small shell language and could certainly support these features, independent of your system’s shell.

[Amin_Yahyaabadi]

@simeonschaub

You can do this by quoting the command with single or double backticks

In some cases (like this simple echo hey/*) this is possible, but this does not work all the time. && was an example. Additionally, going through each command one by one and escaping them is very inconvenient.

@simeonschaub

You usually don’t just want to run arbitrary strings as commands.

Wanting to run a command using a simple String is not a rare request. Among the languages I know, only PowerShell and Julia have weird behaviors for running external commands. PowerShell keeps introducing workarounds for its current broken behavior. Julia tries to stick to the API, but run still does not work properly.

@simeonschaub

Base.shell_parse

As I showed, Base.shell_parse is not stable and a lot of the methods are not defined.

[Tamas_Papp]

@Amin_Yahyaabadi

Wanting to run a command using a simple String is not a rare request.

It may not be, but as explained by @simeonschaub and @giordano above, it is not a well-defined request either, as it depends on shell semantics, which are underspecified (your shell could be literally anything).

If you don’t care about the potential issues, you can always just do something like

bashit(str) = run(`bash -c $str`)

bashit("echo hey && echo bye")

[Amin_Yahyaabadi]

@Tamas_Papp

If you don’t care about the potential issues, you can always just do something like

I understand that this could be feature for those who are interested but the issues that the current behavior brings are more than what a simple run(x::String) has.

For some context, the most used languages don’t have such behavior, and I have not faced such issues in those languages.

# js
const childProcess = require('child_process');
childProcess.execSync("echo hi/* && echo bye")

# python
import os
os.system("echo hi/* && echo bye")

# c++
#include<stdlib.h>

int main() {
   std::system("echo hi/* && echo bye");
   return 0;
}

Probably I can continue with the examples, but you get the idea.

[stevengj]

@Amin_Yahyaabadi

For some context, the most used languages don’t have such behavior, and I have not faced such issues in those languages.

The system command in other languages executes a shell. run in Julia is not a shell — it is a function to directly launch processes (analogous to subprocess.call in Python). By relying on the shell to launch processes you run into lots of other issues (+ portability problems): Shelling Out Sucks … the Python developers also concluded the same thing (PEP 324).

Julia has globbing and pipelines etcetera, but they aren’t implemented with shell syntax. They are implemented in Julia, e.g. pipelines via pipeline(...) and globbing via Glob.jl.

Just don’t expect run to be equivalent to bash. If you want to launch a shell like bash and pass it a string to execute, you can do that in Julia, of course (by spawning the shell explicitly).

[Amin_Yahyaabadi]

I wish “being explicit” was enough, but it does not work always. I hope someday Cmd works like a String so I can use it without applying all these workarounds.

Here is another case that Cmd is broken:

Consider that I want to explicitly run this Julia command:

julia> ex = :( using Printf; @printf("%f",1) )
quote
    using Printf
    #= REPL[31]:1 =#
    #= REPL[31]:1 =# @printf "%f" 1
end

First try

julia> run(`julia -e "$ex"`)
ERROR: LoadError: UndefVarError: @printf not defined
in expression starting at none:4
ERROR: failed process: Process(`julia -e 'begin
    using Printf
    #= REPL[31]:1 =#
    #= REPL[31]:1 =# @printf "%f" 1
end'`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run(::Cmd) at .\process.jl:438
 [4] top-level scope at REPL[32]:1

Another try

julia> run(`julia -e $ex`)
ERROR: LoadError: UndefVarError: @printf not defined
in expression starting at none:4
ERROR: failed process: Process(`julia -e 'begin
    using Printf
    #= REPL[31]:1 =#
    #= REPL[31]:1 =# @printf "%f" 1
end'`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run(::Cmd) at .\process.jl:438
 [4] top-level scope at REPL[35]:1

Trying to hack into Julia AST to make this work!

julia> ex = Expr(:toplevel, :(using Printf), quote
           @printf("%f",1)
       end)
:($(Expr(:toplevel, :(using Printf), quote
    #= REPL[38]:2 =#
    #= REPL[38]:2 =# @printf "%f" 1
end)))

julia> run(`julia -e $ex`)  # same as  run(`julia -e "$ex"`)
ERROR: syntax: "$" expression outside quote
Stacktrace:
 [1] top-level scope at none:1
ERROR: failed process: Process(`julia -e '$(Expr(:toplevel, :(using Printf), quote
    #= REPL[38]:2 =#
    #= REPL[38]:2 =# @printf "%f" 1
end))'`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run(::Cmd) at .\process.jl:438
 [4] top-level scope at REPL[39]:1

Just give up and introduce another workaround:

julia> Cmd_fixer(expr::Expr) = replace(string(expr), r"^begin([\s\S]*)end$"=>s"\1");

julia> ex = :( using Printf; @printf("%f",1) );

julia> run(`julia -e $(Cmd_fixer(ex) )`)
1.000000Process(`julia -e '
    using Printf
    #= REPL[45]:1 =#
    #= REPL[45]:1 =# @printf "%f" 1
'`, ProcessExited(0))

[anon92994695]

@Amin_Yahyaabadi - I don’t understand the actual problem here. Is there a reason why you are receiving julia commands and not strings which you want to toss into a REPL? It seems a little tricky to do what you’re doing.
while the following:

ex = "using Printf; @printf(\"%f\",1)"
"using Printf; @printf(\"%f\",1)"

julia> run(`julia -e $ex`)
1.000000Process(`julia -e 'using Printf; @printf("%f",1)'`, ProcessExited(0))

is really easy :).

[Amin_Yahyaabadi]

Calling external commands is very common in general. Here, I was just showing a case that specifying the program explicitly does not help.

If your question is about the exact application of this, you would do this when you want to run a command outside of your current session. For example, I used external Julia processes in SnoopCompileBot to snoop the compilation of programs.

About your solution, what if you could not escape the strings manually? Taking your solution with @Tamas_Papp’s suggestion:

julia> ex= raw"""using Printf; @printf("%f",1)"""
"using Printf; @printf(\"%f\",1)"

julia> bashit(str) = run(`bash -c $str`);

julia>  bashit("julia -e $ex")
/bin/bash: -c: line 0: syntax error near unexpected token `"%f",1'
/bin/bash: -c: line 0: `julia -e using Printf; @printf("%f",1)'
ERROR: failed process: Process(`bash -c 'julia -e using Printf; @printf("%f",1)'`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] bashit(::String) at .\REPL[12]:1
 [5] top-level scope at REPL[13]:1

julia> bashit("julia -e '$ex'")
/bin/bash: julia: command not found
ERROR: failed process: Process(`bash -c "julia -e 'using Printf; @printf(\"%f\",1)'"`, ProcessExited(127)) [127]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] bashit(::String) at .\REPL[2]:1
 [5] top-level scope at REPL[3]:1

Try cmd:

julia> run(`cmd /c 'julia -e $ex'`)
ERROR: syntax: "$" expression outside quote
Stacktrace:
 [1] top-level scope at none:1
ERROR: failed process: Process(`cmd /c 'julia -e $ex'`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run(::Cmd) at .\process.jl:438

julia> run(`cmd /c "julia -e $ex"`)
ERROR: syntax: incomplete: premature end of input
Stacktrace:
 [1] top-level scope at none:1
ERROR: failed process: Process(`cmd /c 'julia -e using Printf; @printf("%f",1)'`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run(::Cmd) at .\process.jl:438
 [4] top-level scope at REPL[3]:1

Trying some Nodejs code:

julia>  ex= raw"""node -e 'console.log("x")' """
"node -e 'console.log(\"x\")' "

using bashit

julia> bashit(str) = run(`bash -c $ex`);

julia> bashit(ex)
/bin/bash: node: command not found
ERROR: failed process: Process(`bash -c "node -e 'console.log(\"x\")' "`, ProcessExited(127)) [127]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] bashit(::String) at .\REPL[13]:1
 [5] top-level scope at REPL[14]:1


julia> run(`node -v`) # but node is there
v14.7.0
Process(`node -v`, ProcessExited(0))

Try cmd

julia> run(`cmd /c "$ex"`) # x is not printed
Process(`cmd /c "node -e 'console.log(\"x\")' "`, ProcessExited(0))

The point is that for each situation a different workaround is required and. Keeping track of these in a real world application, which you need to run multiple external commands, this becomes frustrating.

[giordano]

@Amin_Yahyaabadi

The point is that for each situation a different workaround is required

What I see here is that having to deal with each underlying shell is a pain. Which is the whole point of not relying on it.

[anon92994695]

i see what you’re saying now. Normally what I would do in these instances is, write to disk and run the code. But I think I’ve only done that once. Not sure what the generic solution is. Just trying to understand the problem.

[Amin_Yahyaabadi]

Yes. I ended up deprecating macros in SnoopCompileBot and only accept the code that is written to the disk. However, I still use Cmd_fixer for interpolating my code into the external Julia process.

[kristoffer.carlsson]

@Amin_Yahyaabadi

Currently, in Julia the run function is not as stable as the rest of the Julia.

Did you read Running External Programs · The Julia Language. It seems a lot of the issues here are due to a misunderstanding of what run actually does. The docs explicitly discusses some of the issues you encounter here.

[Tamas_Papp]

@Amin_Yahyaabadi

using bashit

julia> bashit(str) = run(`bash -c $ex`);

julia> bashit(ex)
/bin/bash: node: command not found

Note that this function is just a suggestion, not a well-tested and robust implementation of anything.

It is still not clear to me what you want Julia to do here — what you call “external commands” is actually code written in some shell syntax. You did not say which shell you want, so I imagine you are assuming they are the same, but in fact even simple globbing has different behavior in various shells in some corner cases.

A shell is a special kind of programming language, run does not anything like that. But most of that functionality is available in other ways in Julia. Instead of invoking the shell, I would recommend just coding a solution in Julia.

[Amin_Yahyaabadi]

Yes, and many of the examples don’t work on Windows unless you modify them. There is no note on how to run these commands on any operating system. The examples use things that are not installed on every operating system: grep, perl, rm -rf, cut -d:. As soon as you start to add your cmd /c or bash -c modification, you run into all of the errors I mentioned, and you should apply special workarounds for them.

https://docs.julialang.org/en/v1/manual/running-external-programs/#Running-External-Programs-1

julia> run(`echo hello`);
ERROR: IOError: could not spawn `echo hello`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] #585 at .\process.jl:112 [inlined]
 [3] setup_stdios(::Base.var"#585#586"{Cmd}, ::Array{Any,1}) at .\process.jl:196
 [4] _spawn at .\process.jl:111 [inlined]
 [5] run(::Cmd; wait::Bool) at .\process.jl:439
 [6] run(::Cmd) at .\process.jl:438
 [7] top-level scope at REPL[7]:1

Correct form

julia> run(`cmd /c echo hello`);
hello

https://docs.julialang.org/en/v1/manual/running-external-programs/#Pipelines-1

julia> run(`echo hello` & `echo world`);
ERROR: IOError: could not spawn `echo hello`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] _spawn(::Cmd, ::Array{Any,1}, ::Base.ProcessChain) at .\process.jl:181
 [3] _spawn(::Base.AndCmds, ::Array{Any,1}, ::Base.ProcessChain) at .\process.jl:174
 [4] #587 at .\process.jl:120 [inlined]
 [5] setup_stdios(::Base.var"#587#588"{Base.AndCmds}, ::Array{Any,1}) at .\process.jl:196
 [6] _spawn at .\process.jl:119 [inlined]
 [7] run(::Base.AndCmds; wait::Bool) at .\process.jl:439
 [8] run(::Base.AndCmds) at .\process.jl:438
 [9] top-level scope at REPL[3]:1

Correct form:

julia> run(`cmd /c echo hello` & `cmd /c echo world`);
hello
world

[Amin_Yahyaabadi]

@Tamas_Papp

A shell is a special kind of programming language, run does not anything like that. But most of that functionality is available in other ways in Julia. Instead of invoking the shell, I would recommend just coding a solution in Julia.

That’s the best practice, however, it is inevitable in a real application. Anything that is considered in the General Programming area needs this external interface. For example, to check if your file is tracked using git you would do something like this:

Here, I should try three different solutions until one of them work:

julia> ex = raw""" git ls-files  'Project.toml' | echo """
" git ls-files  'Project.toml' | echo "

julia>  run(pipeline(`basch -c $ex`))
ERROR: IOError: could not spawn `basch -c " git ls-files  'Project.toml' | echo "`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] #585 at .\process.jl:112 [inlined]
 [3] setup_stdios(::Base.var"#585#586"{Cmd}, ::Array{Any,1}) at .\process.jl:196
 [4] _spawn at .\process.jl:111 [inlined]
 [5] run(::Cmd; wait::Bool) at .\process.jl:439
 [6] run(::Cmd) at .\process.jl:438
 [7] top-level scope at REPL[16]:1

julia>  run(`bash -c $ex`)

Process(`bash -c " git ls-files  'Project.toml' | echo "`, ProcessExited(0))

julia>  run(pipeline(`pwsh -c $ex`))
Project.toml
Process(`pwsh -c " git ls-files  'Project.toml' | echo "`, ProcessExited(0))

Probably I can try to make Git.jl compatible with newer Julia, but that is too much work for a simple check.

[PetrKryslUCSD]

@Amin_Yahyaabadi

ex = raw""" git ls-files  'Project.toml' | echo """
" git ls-files  'Project.toml' | echo "

You had a typo in your first command. This works:

julia> ex = raw""" git ls-files  'Project.toml'  """                                                           
" git ls-files  'Project.toml'  "                                                                              
                                                                                                               
julia> run(pipeline(`bash -c "$(ex)"`))                                                                        
Project.toml                                                                                                   
Process(`bash -c " git ls-files  'Project.toml'  "`, ProcessExited(0))   

[Amin_Yahyaabadi]

Yes, sorry. I was trying to make the command runnable on different operating systems (because of grep, so people can try it without issues. This was the original command:

# this checks if Manifest is tracked by Git, and if so it reverts it to the final state
git ls-files 'Manifest.toml' | grep . && git checkout -- 'Manifest.toml'

I finally gave up trying:
https://github.com/aminya/SnoopCompileBot.jl/blob/2bca967ae18c57aacb2028e0a550262c9007befa/src/postprocess.jl#L21