How to PackageCompile ArgParse

New to Julia
1

Hello!

Background

I’m fairly new to julia and love it’s potential, but one thing always seems to slow me down. I write scripts almost exclusively (executables with #!/path/to/julia at the top), and they’re oh-so-very-slow as soon as a package gets involved.

For example, I’m creating a CLI program and am using ArgParse for that. I have this very simple program:

#!/usr/local/bin/julia
using ArgParse

function args_parse()
    s = ArgParseSettings()

    @add_arg_table s begin
        "arg1"
            help = "a positional argument"
            required = true
    end

    return parse_args(s)
end

function main()
    parsed_args = args_parse()
    println("Parsed args:")
    for (arg,val) in parsed_args
        println("  $arg  =>  $val")
    end
end

main()

Here’s what I get when I time a run:

[andromodon@yogie cliProject]$ time /opt/julia/julia-1.2.0/bin/julia ./cliProgram.jl 
required argument arg1 was not provided
usage: cliProgram.jl arg1

real    0m4.300s
user    0m4.324s
sys     0m0.299s

4 seconds!!! If this is in my inner dev loop, which I kinda want it to be able to be, my project would take months instead of weeks to complete. I know there are well known work-arounds (like never closing julia), but those aren’t ideal. Even my end user shouldn’t end up having to wait 4 seconds for the program to parse their command line arguments.

Problem

So, after some googling, I found that PackageCompiler helps in many circumstances. The thing is I can’t get it to work either.

Here’s my setup script:

#!/usr/local/bin/julia

#This script installs julia dependencies, as needed.
using Pkg

Pkg.add(PackageSpec(name="ArgParse", version= "0.6.2"))
Pkg.build("PackageCompiler")

using PackageCompiler
using ArgParse

#Make loading ArgParse faster (hopefully):
compile_incremental(:ArgParse, force=true)

Running that gives this:

[andromodon@yogie cliProject]$ /opt/julia/julia-1.2.0/bin/julia ./setup.jl 
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/.julia/environments/v1.2/Project.toml`
 [no changes]
  Updating `~/.julia/environments/v1.2/Manifest.toml`
 [no changes]
  Building LibCURL ────────→ `~/.julia/packages/LibCURL/lWJxD/deps/build.log`
  Building WinRPM ─────────→ `~/.julia/packages/WinRPM/Y9QdZ/deps/build.log`
  Building PackageCompiler → `~/.julia/packages/PackageCompiler/CJQcs/deps/build.log`
[ Info: Registered package ArgParse, using already given UUID: c7e460c6-2fb9-53a9-8c5b-16f535851c63
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/.julia/packages/PackageCompiler/CJQcs/packages/ArgParse/Project.toml`
  [34da2185] + Compat v2.2.0
  [b718987f] + TextWrap v0.3.0
  Updating `~/.julia/packages/PackageCompiler/CJQcs/packages/ArgParse/Manifest.toml`
  [34da2185] + Compat v2.2.0
  [b718987f] + TextWrap v0.3.0
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8bb1440f] + DelimitedFiles 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [a63ad114] + Mmap 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [1a1011a3] + SharedArrays 
  [6462fe0b] + Sockets 
  [2f01184e] + SparseArrays 
  [10745b16] + Statistics 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 
 Resolving package versions...
  Updating `~/.julia/packages/PackageCompiler/CJQcs/packages/ArgParse/Project.toml`
  [9b87118b] + PackageCompiler v0.6.4
  [44cfe95a] + Pkg 
  Updating `~/.julia/packages/PackageCompiler/CJQcs/packages/ArgParse/Manifest.toml`
  [9e28174c] + BinDeps v0.8.10
  [b99e7846] + BinaryProvider v0.5.8
  [e1450e63] + BufferedStreams v1.0.0
  [0862f596] + HTTPClient v0.2.1
  [b27032c2] + LibCURL v0.5.2
  [522f3ed2] + LibExpat v0.5.0
  [2ec943e9] + Libz v1.0.0
  [9b87118b] + PackageCompiler v0.6.4
  [30578b45] + URIParser v0.4.0
  [c17dfb99] + WinRPM v0.4.2
Activating environment at `~/.julia/packages/PackageCompiler/CJQcs/packages/ArgParse/Project.toml`
ERROR: LoadError: `ArgParse` is a direct dependency, but does not appear in the manifest. If you intend `ArgParse` to be a direct dependency, run `Pkg.resolve()` to populate the manifest. Otherwise, remove `ArgParse` with `Pkg.rm("ArgParse")`. Finally, run `Pkg.instantiate()` again.
Stacktrace:
 [1] pkgerror(::String, ::Vararg{String,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/Types.jl:112
 [2] #instantiate#81(::Nothing, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Pkg.API.instantiate), ::Pkg.Types.Context) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:472
 [3] instantiate at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:461 [inlined]
 [4] #instantiate#80 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:458 [inlined]
 [5] instantiate() at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:458
 [6] top-level scope at /home/andromodon/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl:6
 [7] include at ./boot.jl:328 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1094
 [9] include(::Module, ::String) at ./Base.jl:31
 [10] exec_options(::Base.JLOptions) at ./client.jl:295
 [11] _start() at ./client.jl:464
in expression starting at /home/andromodon/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl:6
ERROR: LoadError: failed process: Process(`/opt/julia/julia-1.2.0/bin/julia --compile=all --optimize=0 -g1 --trace-compile=/home/andromodon/.julia/packages/PackageCompiler/CJQcs/packages/precompile_tmp.jl --history-file=yes --code-coverage=none --inline=yes --math-mode=ieee --handle-signals=yes --warn-overwrite=no --compile=yes --depwarn=yes --cpu-target=native --track-allocation=none --sysimage-native-code=yes --sysimage=/opt/julia/julia-1.2.0/lib/julia/sys.so -g1 --compiled-modules=yes --optimize=2 /home/andromodon/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:813 [inlined]
 [2] #run#536(::Bool, ::typeof(run), ::Cmd) at ./process.jl:728
 [3] run at ./process.jl:726 [inlined]
 [4] #run_julia#1 at /home/andromodon/.julia/packages/PackageCompiler/CJQcs/src/compiler_flags.jl:225 [inlined]
 [5] #run_julia at ./none:0 [inlined]
 [6] snoop(::Symbol, ::String, ::String, ::String, ::Bool, ::Array{Any,1}) at /home/andromodon/.julia/packages/PackageCompiler/CJQcs/src/snooping.jl:33
 [7] (::getfield(PackageCompiler, Symbol("##35#37")){Array{Any,1},Tuple{Symbol},Dict{Any,Any},String,Dict{String,Array{Dict{String,Any},1}}})(::IOStream) at /home/andromodon/.julia/packages/PackageCompiler/CJQcs/src/snooping.jl:123
 [8] #open#312(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(open), ::getfield(PackageCompiler, Symbol("##35#37")){Array{Any,1},Tuple{Symbol},Dict{Any,Any},String,Dict{String,Array{Dict{String,Any},1}}}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:375
 [9] open at ./iostream.jl:373 [inlined]
 [10] #snoop_packages#34 at /home/andromodon/.julia/packages/PackageCompiler/CJQcs/src/snooping.jl:109 [inlined]
 [11] #snoop_packages at ./none:0 [inlined]
 [12] #compile_incremental#63 at /home/andromodon/.julia/packages/PackageCompiler/CJQcs/src/incremental.jl:176 [inlined]
 [13] (::getfield(PackageCompiler, Symbol("#kw##compile_incremental")))(::NamedTuple{(:force,),Tuple{Bool}}, ::typeof(compile_incremental), ::Symbol) at ./none:0
 [14] top-level scope at /home/andromodon/tmp/cliProject/setup.jl:13
 [15] include at ./boot.jl:328 [inlined]
 [16] include_relative(::Module, ::String) at ./loading.jl:1094
 [17] include(::Module, ::String) at ./Base.jl:31
 [18] exec_options(::Base.JLOptions) at ./client.jl:295
 [19] _start() at ./client.jl:464
in expression starting at /home/andromodon/tmp/cliProject/setup.jl:13

I’m stuck and am not quite sure to do. I read some threads about changing a line in a package or something but I shouldn’t have to edit packages myself to get a basic CLI working quickly with julia. What am I missing??

BTW, I’m using Julia 1.2 since when I was using 1.3 I suffered from this issue.

Thanks for your help. :slight_smile:

1 Like
2

offtopic comment, but ArgParse should be included in stdlibs and ship with julia.

3

I’ ve PackageCompiled my script with ArgParse before. The speed was faster, but still not ideal, like 20s down to 5s. I ended up with writing a golang cmdline parser that sends arguments to my forever running julia webserver, after which everything finishes within 0.1s.

4

You could also try :
https://github.com/KristofferC/PackageCompilerX.jl
It worked for me.

1 Like
5

Just thought about if it is possible to use pycall to use a python arg parser, have anyone tried?

6

I’ve been trying to get the same thing working. I think you’re better off setting it up as a project.

I’m not really sure if this setup is ideal, but it at least works.

First create your project.

julia -e 'using Pkg; Pkg.generate("MyProj")'
cd MyProj

You’ll need to setup a precompile script (see here for details on precompiling). Unfortunately you can’t really run your main script as your precompile script since ArgParse errors out when passed invalid arguments. So you either need to not run parse_args or create some sort of noop command.

So your main app function will look something like this.

/MyProj/src/MyProj.jl

module MyProj

export run

using ArgParse

function run(parseargs::Bool)
    s = ArgParseSettings()

    @add_arg_table s begin
        "arg1"
            help = "a positional argument"
            required = true
    end

    # don't run parse_args when precompiling.
    # alternatively you could just create a noop command
    # and pass ARGS to the run function.
    if !parseargs
        return
    end

    args = parse_args(ARGS, s)
    println(args)
end

end # module

/MyProj/app.jl

using MyProj
MyProj.run(true)

Now you just create the precompile and sysimage creation scripts.

/MyProj/scripts/precompile.js

using MyProj
MyProj.run(false) # same as app.jl but does not run parse_args()

/MyProj/scripts/create_image.jl

using PackageCompiler
using Pkg
Pkg.activate(".")
create_sysimage([:ArgParse, :MyProj];
    sysimage_path="MyProj.so",
    precompile_execution_file=joinpath(@__DIR__, "precompile.jl")
)

Then to setup the sysimage run the script.

julia scripts/create_image.jl

And then you can run your program

time julia --project=@. -JMyProj.so app.jl arg1 -h

real	0m0.640s
user	0m0.770s
sys	    0m0.110s

See here a complete example.

4 Likes