Determining whether julia is "self-built" or "shipped"

I have an application whose build using PackageCompiler fails strangely on my workstation when using cpu_target="x86_64" (see Understanding runtime errors with PackageCompiler built executables).

If I understand correctly this comment, the cpu_target="x86_64" option is not compatible when julia is “self built”, which is indeed the case on my workstation (I compiled it locally).

However, I’m also trying to compile it in a debian-based singularity container, that has a “shipped julia” installed. The goal is to be able to run the application “anywhere” (where “anywhere” currently means “on a computing cluster with no julia installation” and where the user should not need to bother with anything julia related, just loading singularity and running the container should work). For this, I need to cpu_target="x86_64" to be set.

Is there a way (either from the julia build script itself or from the OS) to determine whether julia is “self-build” or “shipped”, in order to adapt the compiling option automatically, without having to manually edit the script whenever the host system changes?

I hope I am not getting this totally wrong…
On your workstation you are building Julia from source.

However in the container build you should be pulling Julia won as a ‘shipped’ piece of software from the Julialang site. Am I understanding this correctly?

This may help: ANN: JILL - Julia Installer 4 Linux (and MacOS) - Light

Thanks for your suggestion, but there is actually no issue with how to get julia in the container. I’m using a container that already has julia in it: https://hub.docker.com/_/julia

The julia version in the container is compatible with the cpu_target="x86_64" option of PackageCompiler, and that’s what I actually need.

But for debugging purposes, I want to be able to use PackageCompiler on my workstation. The build is done via deps/build.jl, which looks as follows:

import Pkg
println("Building qaf_demux")
Pkg.add("REPL")
Pkg.add("PackageCompiler")
using PackageCompiler
# setting cpu_target does not work with a self-built Julia: https://github.com/NHDaly/ApplicationBuilder.jl/issues/62#issuecomment-503721859
# build_executable(joinpath(@__DIR__, "../bin/qaf_demux_to_compile.jl"), "qaf_demux", snoopfile=joinpath(@__DIR__, "../bin/snoop.jl"))
build_executable(joinpath(@__DIR__, "../bin/qaf_demux_to_compile.jl"), "qaf_demux", snoopfile=joinpath(@__DIR__, "../bin/snoop.jl"), cpu_target="x86_64")

This is all under version control, and whenever I want to try to build the application, I need to manually change which line is commented and which is not at the end of the above script. This in turn makes me see a modified file under git, but this is just for local use and is not meant to be committed.

What I would like is a way to have a build.jl that works both on my local workstation and in the container, at build time.

Something like that:

import Pkg
println("Building qaf_demux")
Pkg.add("REPL")
Pkg.add("PackageCompiler")
using PackageCompiler
# setting cpu_target does not work with a self-built Julia: https://github.com/NHDaly/ApplicationBuilder.jl/issues/62#issuecomment-503721859
if (the julia currently running this code is locally compiled)
    build_executable(joinpath(@__DIR__, "../bin/qaf_demux_to_compile.jl"), "qaf_demux", snoopfile=joinpath(@__DIR__, "../bin/snoop.jl"))
else
    build_executable(joinpath(@__DIR__, "../bin/qaf_demux_to_compile.jl"), "qaf_demux", snoopfile=joinpath(@__DIR__, "../bin/snoop.jl"), cpu_target="x86_64")
end

Maybe I should explicit what is meant by “shipped” here. This is the vocabulary I took from the linked comment in my original post:

Ah, sorry, it’s because --cpu-target=x86-64 is incompatible with self-built julia. You can either:

  1. Use a shipped julia binary, or
  2. Change the cpu target to cpu_target="native" in build_app_bundle(). But If you do this, then your app will be significantly less portable to other machines (it will only run on other machines with identical cpus).

Why don’t you use the same target as julia itself uses when building releases which is generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)? Julia can load optimized code based on runtime cpu feature detection but the generic fallback should allow it to work on most cpus.

These options work when building the application in the container (are they supposed to be “better” than a plain and general x86_64, but still very portable ?), but not on the local workstation. I get the very weird error again:

┌ Error: Error building `QafDemux`: 
│ [ Info: Smallest distance between barcodes: 3
│ ┌ Info: Fastq files have been written:
│ │   outfile_paths_dict =
│ │    Dict{String,String} with 13 entries:
│ │      "GCAGAGAGGAAT" => "test_run/GCAGAGAGGAAT.fastq.gz"
│ │      "GCAGAGAGAGAC" => "test_run/GCAGAGAGAGAC.fastq.gz"
│ │      "GCAGAGATGTTG" => "test_run/GCAGAGATGTTG.fastq.gz"
│ │      "GCAGAGACCAAC" => "test_run/GCAGAGACCAAC.fastq.gz"
│ │      "Undetermined" => "test_run/Undetermined.fastq.gz"
│ │      "GCAGAGAGGCTA" => "test_run/GCAGAGAGGCTA.fastq.gz"
│ │      "GCAGAGACAACT" => "test_run/GCAGAGACAACT.fastq.gz"
│ └      ⋮              => ⋮
│ ┌ Warning: Snoop file errored. Precompile statements were recorded untill error!
│ │   exception =
│ │    LoadError: MethodError: no method matching make_record_reader(::SubString{String}, ::Array{String,1}, ::getfield(QafDemux, Symbol("#seq_qual_extractor#7")){UnitRange{Int64}}, ::Bool)
│ │    Closest candidates are:
│ │      make_record_reader(!Matched::String, ::Array{String,1}, ::Any, ::Any) at /home/bli/src/qaf_demux/Julia/QafDemux/src/QafDemux.jl:208
│ │      make_record_reader(!Matched::String, ::Array{String,1}, ::Any) at /home/bli/src/qaf_demux/Julia/QafDemux/src/QafDemux.jl:208
│ │    in expression starting at /home/bli/src/qaf_demux/Julia/QafDemux/bin/snoop.jl:8
│ └ @ Main ~/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl:7
│ ERROR: Unable to find compatible target in system image.
│ [ Info: used 267 out of 267 precompile statements
│ ERROR: LoadError: failed process: Process(`/home/bli/src/julia/usr/bin/julia '--cpu-target=generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)' --output-o=qaf_demux.a --track-allocation=none --code-coverage=none --inline=yes --math-mode=ieee --startup-file=no --compile=yes --track-allocation=none --sysimage-native-code=yes --sysimage=/home/bli/src/julia/usr/lib/julia/sys.so --compiled-modules=yes --optimize=0 /home/bli/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]
│ 
│ Stacktrace:
│  [1] pipeline_error at ./process.jl:813 [inlined]
│  [2] #run#536(::Bool, ::typeof(run), ::Cmd) at ./process.jl:728
│  [3] run at ./process.jl:726 [inlined]
│  [4] #run_julia#1 at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/compiler_flags.jl:225 [inlined]
│  [5] #run_julia at ./none:0 [inlined]
│  [6] (::getfield(PackageCompiler, Symbol("##13#14")){Base.Iterators.Pairs{Symbol,Any,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),Tuple{Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,String,Nothing,Nothing,Nothing,Nothing,Nothing}}},String})() at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:263
│  [7] cd(::getfield(PackageCompiler, Symbol("##13#14")){Base.Iterators.Pairs{Symbol,Any,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),Tuple{Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,String,Nothing,Nothing,Nothing,Nothing,Nothing}}},String}, ::String) at ./file.jl:96
│  [8] #build_object#12(::Base.Iterators.Pairs{Symbol,Any,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),Tuple{Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,String,Nothing,Nothing,Nothing,Nothing,Nothing}}}, ::typeof(PackageCompiler.build_object), ::String, ::String, ::String, ::Bool) at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:262
│  [9] #build_object at ./none:0 [inlined]
│  [10] build_object(::String, ::String, ::String, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::String, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing) at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:241
│  [11] #static_julia#5(::Nothing, ::Bool, ::Bool, ::Nothing, ::String, ::String, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::String, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::typeof(static_julia), ::String) at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:162
│  [12] #static_julia at ./tuple.jl:0 [inlined]
│  [13] #build_executable#31 at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/api.jl:104 [inlined]
│  [14] #build_executable at ./none:0 [inlined] (repeats 2 times)
│  [15] top-level scope at /home/bli/src/qaf_demux/Julia/QafDemux/deps/build.jl:8
│  [16] include at ./boot.jl:328 [inlined]
│  [17] include_relative(::Module, ::String) at ./loading.jl:1094
│  [18] include(::Module, ::String) at ./Base.jl:31
│  [19] include(::String) at ./client.jl:431
│  [20] top-level scope at none:5
│ in expression starting at /home/bli/src/qaf_demux/Julia/QafDemux/deps/build.jl:8
│ Building qaf_demux
│   Updating registry at `~/.julia/registries/BioJuliaRegistry`
│   Updating git-repo `https://github.com/BioJulia/BioJuliaRegistry.git`
    Updating registry at `~/.julia/registries/General`
│   Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
│   Updating `/tmp/jl_tRerXx/Project.toml`
│   [3fa0cd96] + REPL 
│   Updating `/tmp/jl_tRerXx/Manifest.toml`
│   [2a0f44e3] ~ Base64  [`@stdlib/Base64`] ⇒ 
│   [ade2ca70] ~ Dates  [`@stdlib/Dates`] ⇒ 
│   [8bb1440f] ~ DelimitedFiles  [`@stdlib/DelimitedFiles`] ⇒ 
│   [8ba89e20] ~ Distributed  [`@stdlib/Distributed`] ⇒ 
│   [b77e0a4c] ~ InteractiveUtils  [`@stdlib/InteractiveUtils`] ⇒ 
│   [76f85450] ~ LibGit2  [`@stdlib/LibGit2`] ⇒ 
│   [8f399da3] ~ Libdl  [`@stdlib/Libdl`] ⇒ 
│   [37e2e46d] ~ LinearAlgebra  [`@stdlib/LinearAlgebra`] ⇒ 
│   [56ddb016] ~ Logging  [`@stdlib/Logging`] ⇒ 
│   [d6f4376e] ~ Markdown  [`@stdlib/Markdown`] ⇒ 
│   [a63ad114] ~ Mmap  [`@stdlib/Mmap`] ⇒ 
│   [de0858da] ~ Printf  [`@stdlib/Printf`] ⇒ 
│   [3fa0cd96] ~ REPL  [`@stdlib/REPL`] ⇒ 
│   [9a3f8284] ~ Random  [`@stdlib/Random`] ⇒ 
│   [ea8e919c] ~ SHA  [`@stdlib/SHA`] ⇒ 
│   [9e88b42a] ~ Serialization  [`@stdlib/Serialization`] ⇒ 
│   [1a1011a3] ~ SharedArrays  [`@stdlib/SharedArrays`] ⇒ 
│   [6462fe0b] ~ Sockets  [`@stdlib/Sockets`] ⇒ 
│   [2f01184e] ~ SparseArrays  [`@stdlib/SparseArrays`] ⇒ 
│   [10745b16] ~ Statistics  [`@stdlib/Statistics`] ⇒ 
│   [8dfed614] ~ Test  [`@stdlib/Test`] ⇒ 
│   [cf7118a7] ~ UUIDs  [`@stdlib/UUIDs`] ⇒ 
│   [4ec0a83e] ~ Unicode  [`@stdlib/Unicode`] ⇒ 
│  Resolving package versions...
│   Updating `/tmp/jl_tRerXx/Project.toml`
│  [no changes]
│   Updating `/tmp/jl_tRerXx/Manifest.toml`
│  [no changes]
│ Julia program file:
│   "/home/bli/src/qaf_demux/Julia/QafDemux/bin/qaf_demux_to_compile.jl"
│ C program file:
│   "/home/bli/.julia/packages/PackageCompiler/CJQcs/examples/program.c"
│ Build directory:
│   "/home/bli/src/qaf_demux/Julia/QafDemux/deps/builddir"
└ @ Pkg.Operations ~/src/julia/usr/share/julia/stdlib/v1.2/Pkg/src/backwards_compatible_isolation.jl:647

Maybe the local version of Julia really can’t use the cpu_target option at all.

What if you just fix make_record_reader to take an AbstractString as the first argument?

Looking at https://gitlab.pasteur.fr/bli/qaf_demux/blob/master/Julia/QafDemux/bin/snoop.jl#L8 you pass in a Vector{SubString} to main.

But I see the ERROR: Unable to find compatible target in system image. now which I guess is the real problem.

I then get another error: LoadError: MethodError: no method matching chosetranscoder(::SubString{String})

This too can probably fixed by changing type signatures of my functions, but how come this cpu-target compiling option interacts with allowed type signatures in my code?