# Determining whether julia is "self-built" or "shipped"

**URL:** <https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404>\
**Category:** New to Julia\
**Created:** [October 2, 2019, 12:55pm UTC](https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404 "2019-10-02T12:55:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![blaiseli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/blaiseli/32/10332_2.png) [@blaiseli](https://discourse.julialang.org/u/blaiseli)\
**Post date:** [October 2, 2019, 12:55pm UTC](https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404/1 "2019-10-02T12:55:26Z")

</div>

I have an application whose build using `PackageCompiler` fails strangely on my workstation when using `cpu_target="x86_64"` (see [Understanding runtime errors with PackageCompiler built executables - #5 by blaiseli](https://discourse.julialang.org/t/understanding-runtime-errors-with-packagecompiler-built-executables/29120/5)).

If I understand correctly [this comment](https://github.com/NHDaly/ApplicationBuilder.jl/issues/62#issuecomment-503721859), the `cpu_target="x86_64"` option is not compatible when julia is “self built”, which is indeed the case on my workstation (I compiled it locally).

However, I’m also trying to compile it in a debian-based singularity container, that has a “shipped julia” installed. The goal is to be able to run the application “anywhere” (where “anywhere” currently means “on a computing cluster with no julia installation” and where the user should not need to bother with anything julia related, just loading singularity and running the container should work). For this, I need to `cpu_target="x86_64"` to be set.

Is there a way (either from the julia build script itself or from the OS) to determine whether julia is “self-build” or “shipped”, in order to adapt the compiling option automatically, without having to manually edit the script whenever the host system changes?

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 2, 2019, 4:21pm UTC](https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404/2 "2019-10-02T16:21:13Z")

</div>

I hope I am not getting this totally wrong…  
On your workstation you are building Julia from source.

However in the container build you should be pulling Julia won as a ‘shipped’ piece of software from the Julialang site. Am I understanding this correctly?

This may help: [ANN: JILL - Julia Installer 4 Linux (and MacOS) - Light](https://discourse.julialang.org/t/ann-jill-julia-installer-4-linux-and-macos-light/27234)

---

<div class="post-metadata">

**Author:** ![blaiseli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/blaiseli/32/10332_2.png) [@blaiseli](https://discourse.julialang.org/u/blaiseli)\
**Post date:** [October 2, 2019, 4:41pm UTC](https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404/3 "2019-10-02T16:41:24Z")

</div>

Thanks for your suggestion, but there is actually no issue with how to get julia in the container. I’m using a container that already has julia in it: [Docker](https://hub.docker.com/_/julia)

The julia version in the container is compatible with the `cpu_target="x86_64"` option of `PackageCompiler`, and that’s what I actually need.

But for debugging purposes, I want to be able to use `PackageCompiler` on my workstation. The build is done via `deps/build.jl`, which looks as follows:

```julia
import Pkg
println("Building qaf_demux")
Pkg.add("REPL")
Pkg.add("PackageCompiler")
using PackageCompiler
# setting cpu_target does not work with a self-built Julia: https://github.com/NHDaly/ApplicationBuilder.jl/issues/62#issuecomment-503721859
# build_executable(joinpath(@ __DIR__ , "../bin/qaf_demux_to_compile.jl"), "qaf_demux", snoopfile=joinpath(@ __DIR__ , "../bin/snoop.jl"))
build_executable(joinpath(@ __DIR__ , "../bin/qaf_demux_to_compile.jl"), "qaf_demux", snoopfile=joinpath(@ __DIR__ , "../bin/snoop.jl"), cpu_target="x86_64")

```

This is all under version control, and whenever I want to try to build the application, I need to manually change which line is commented and which is not at the end of the above script. This in turn makes me see a modified file under git, but this is just for local use and is not meant to be committed.

What I would like is a way to have a `build.jl` that works both on my local workstation and in the container, at build time.

Something like that:

```julia
import Pkg
println("Building qaf_demux")
Pkg.add("REPL")
Pkg.add("PackageCompiler")
using PackageCompiler
# setting cpu_target does not work with a self-built Julia: https://github.com/NHDaly/ApplicationBuilder.jl/issues/62#issuecomment-503721859
if (the julia currently running this code is locally compiled)
    build_executable(joinpath(@ __DIR__ , "../bin/qaf_demux_to_compile.jl"), "qaf_demux", snoopfile=joinpath(@ __DIR__ , "../bin/snoop.jl"))
else
    build_executable(joinpath(@ __DIR__ , "../bin/qaf_demux_to_compile.jl"), "qaf_demux", snoopfile=joinpath(@ __DIR__ , "../bin/snoop.jl"), cpu_target="x86_64")
end

```

Maybe I should explicit what is meant by “shipped” here. This is the vocabulary I took from the linked comment in my original post:

> Ah, sorry, it’s because `--cpu-target=x86-64` is incompatible with self-built julia. You can either:
> 
> 1. Use a shipped julia binary, or
> 2. Change the cpu target to `cpu_target="native"` in build\_app\_bundle(). But If you do this, then your app will be significantly less portable to other machines (it will only run on other machines with identical cpus).

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [October 2, 2019, 4:47pm UTC](https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404/4 "2019-10-02T16:47:53Z")

</div>

Why don’t you use the same target as julia itself uses when building releases which is `generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)`? Julia can load optimized code based on runtime cpu feature detection but the `generic` fallback should allow it to work on most cpus.

---

<div class="post-metadata">

**Author:** ![blaiseli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/blaiseli/32/10332_2.png) [@blaiseli](https://discourse.julialang.org/u/blaiseli)\
**Post date:** [October 2, 2019, 5:12pm UTC](https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404/5 "2019-10-02T17:12:25Z")

</div>

> [@kristoffer.carlsson](#):
>
> generic;sandybridge,-xsaveopt,clone\_all;haswell,-rdrnd,base(1)

These options work when building the application in the container (are they supposed to be “better” than a plain and general `x86_64`, but still very portable ?), but not on the local workstation. I get the very weird error again:

```julia
┌ Error: Error building `QafDemux`: 
│ [ Info: Smallest distance between barcodes: 3
│ ┌ Info: Fastq files have been written:
│ │ outfile_paths_dict =
│ │ Dict{String,String} with 13 entries:
│ │ "GCAGAGAGGAAT" => "test_run/GCAGAGAGGAAT.fastq.gz"
│ │ "GCAGAGAGAGAC" => "test_run/GCAGAGAGAGAC.fastq.gz"
│ │ "GCAGAGATGTTG" => "test_run/GCAGAGATGTTG.fastq.gz"
│ │ "GCAGAGACCAAC" => "test_run/GCAGAGACCAAC.fastq.gz"
│ │ "Undetermined" => "test_run/Undetermined.fastq.gz"
│ │ "GCAGAGAGGCTA" => "test_run/GCAGAGAGGCTA.fastq.gz"
│ │ "GCAGAGACAACT" => "test_run/GCAGAGACAACT.fastq.gz"
│ └ ⋮ => ⋮
│ ┌ Warning: Snoop file errored. Precompile statements were recorded untill error!
│ │ exception =
│ │ LoadError: MethodError: no method matching make_record_reader(::SubString{String}, ::Array{String,1}, ::getfield(QafDemux, Symbol("#seq_qual_extractor#7")){UnitRange{Int64}}, ::Bool)
│ │ Closest candidates are:
│ │ make_record_reader(!Matched::String, ::Array{String,1}, ::Any, ::Any) at /home/bli/src/qaf_demux/Julia/QafDemux/src/QafDemux.jl:208
│ │ make_record_reader(!Matched::String, ::Array{String,1}, ::Any) at /home/bli/src/qaf_demux/Julia/QafDemux/src/QafDemux.jl:208
│ │ in expression starting at /home/bli/src/qaf_demux/Julia/QafDemux/bin/snoop.jl:8
│ └ @ Main ~/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl:7
│ ERROR: Unable to find compatible target in system image.
│ [ Info: used 267 out of 267 precompile statements
│ ERROR: LoadError: failed process: Process(`/home/bli/src/julia/usr/bin/julia '--cpu-target=generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)' --output-o=qaf_demux.a --track-allocation=none --code-coverage=none --inline=yes --math-mode=ieee --startup-file=no --compile=yes --track-allocation=none --sysimage-native-code=yes --sysimage=/home/bli/src/julia/usr/lib/julia/sys.so --compiled-modules=yes --optimize=0 /home/bli/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]
│ 
│ Stacktrace:
│ [1] pipeline_error at ./process.jl:813 [inlined]
│ [2] #run#536(::Bool, ::typeof(run), ::Cmd) at ./process.jl:728
│ [3] run at ./process.jl:726 [inlined]
│ [4] #run_julia#1 at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/compiler_flags.jl:225 [inlined]
│ [5] #run_julia at ./none:0 [inlined]
│ [6] (::getfield(PackageCompiler, Symbol("##13#14")){Base.Iterators.Pairs{Symbol,Any,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),Tuple{Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,String,Nothing,Nothing,Nothing,Nothing,Nothing}}},String})() at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:263
│ [7] cd(::getfield(PackageCompiler, Symbol("##13#14")){Base.Iterators.Pairs{Symbol,Any,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),Tuple{Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,String,Nothing,Nothing,Nothing,Nothing,Nothing}}},String}, ::String) at ./file.jl:96
│ [8] #build_object#12(::Base.Iterators.Pairs{Symbol,Any,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),Tuple{Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,String,Nothing,Nothing,Nothing,Nothing,Nothing}}}, ::typeof(PackageCompiler.build_object), ::String, ::String, ::String, ::Bool) at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:262
│ [9] #build_object at ./none:0 [inlined]
│ [10] build_object(::String, ::String, ::String, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::String, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing) at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:241
│ [11] #static_julia#5(::Nothing, ::Bool, ::Bool, ::Nothing, ::String, ::String, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::String, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::typeof(static_julia), ::String) at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:162
│ [12] #static_julia at ./tuple.jl:0 [inlined]
│ [13] #build_executable#31 at /home/bli/.julia/packages/PackageCompiler/CJQcs/src/api.jl:104 [inlined]
│ [14] #build_executable at ./none:0 [inlined] (repeats 2 times)
│ [15] top-level scope at /home/bli/src/qaf_demux/Julia/QafDemux/deps/build.jl:8
│ [16] include at ./boot.jl:328 [inlined]
│ [17] include_relative(::Module, ::String) at ./loading.jl:1094
│ [18] include(::Module, ::String) at ./Base.jl:31
│ [19] include(::String) at ./client.jl:431
│ [20] top-level scope at none:5
│ in expression starting at /home/bli/src/qaf_demux/Julia/QafDemux/deps/build.jl:8
│ Building qaf_demux
│ Updating registry at `~/.julia/registries/BioJuliaRegistry`
│ Updating git-repo `https://github.com/BioJulia/BioJuliaRegistry.git`
    Updating registry at `~/.julia/registries/General`
│ Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
│ Updating `/tmp/jl_tRerXx/Project.toml`
│ [3fa0cd96] + REPL 
│ Updating `/tmp/jl_tRerXx/Manifest.toml`
│ [2a0f44e3] ~ Base64 [`@stdlib/Base64`] ⇒ 
│ [ade2ca70] ~ Dates [`@stdlib/Dates`] ⇒ 
│ [8bb1440f] ~ DelimitedFiles [`@stdlib/DelimitedFiles`] ⇒ 
│ [8ba89e20] ~ Distributed [`@stdlib/Distributed`] ⇒ 
│ [b77e0a4c] ~ InteractiveUtils [`@stdlib/InteractiveUtils`] ⇒ 
│ [76f85450] ~ LibGit2 [`@stdlib/LibGit2`] ⇒ 
│ [8f399da3] ~ Libdl [`@stdlib/Libdl`] ⇒ 
│ [37e2e46d] ~ LinearAlgebra [`@stdlib/LinearAlgebra`] ⇒ 
│ [56ddb016] ~ Logging [`@stdlib/Logging`] ⇒ 
│ [d6f4376e] ~ Markdown [`@stdlib/Markdown`] ⇒ 
│ [a63ad114] ~ Mmap [`@stdlib/Mmap`] ⇒ 
│ [de0858da] ~ Printf [`@stdlib/Printf`] ⇒ 
│ [3fa0cd96] ~ REPL [`@stdlib/REPL`] ⇒ 
│ [9a3f8284] ~ Random [`@stdlib/Random`] ⇒ 
│ [ea8e919c] ~ SHA [`@stdlib/SHA`] ⇒ 
│ [9e88b42a] ~ Serialization [`@stdlib/Serialization`] ⇒ 
│ [1a1011a3] ~ SharedArrays [`@stdlib/SharedArrays`] ⇒ 
│ [6462fe0b] ~ Sockets [`@stdlib/Sockets`] ⇒ 
│ [2f01184e] ~ SparseArrays [`@stdlib/SparseArrays`] ⇒ 
│ [10745b16] ~ Statistics [`@stdlib/Statistics`] ⇒ 
│ [8dfed614] ~ Test [`@stdlib/Test`] ⇒ 
│ [cf7118a7] ~ UUIDs [`@stdlib/UUIDs`] ⇒ 
│ [4ec0a83e] ~ Unicode [`@stdlib/Unicode`] ⇒ 
│ Resolving package versions...
│ Updating `/tmp/jl_tRerXx/Project.toml`
│ [no changes]
│ Updating `/tmp/jl_tRerXx/Manifest.toml`
│ [no changes]
│ Julia program file:
│ "/home/bli/src/qaf_demux/Julia/QafDemux/bin/qaf_demux_to_compile.jl"
│ C program file:
│ "/home/bli/.julia/packages/PackageCompiler/CJQcs/examples/program.c"
│ Build directory:
│ "/home/bli/src/qaf_demux/Julia/QafDemux/deps/builddir"
└ @ Pkg.Operations ~/src/julia/usr/share/julia/stdlib/v1.2/Pkg/src/backwards_compatible_isolation.jl:647

```

Maybe the local version of Julia really can’t use the `cpu_target` option at all.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [October 2, 2019, 5:16pm UTC](https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404/6 "2019-10-02T17:16:14Z")

</div>

What if you just fix `make_record_reader` to take an `AbstractString` as the first argument?

Looking at [Julia/QafDemux/bin/snoop.jl · master · Blaise LI / qaf\_demux · GitLab](https://gitlab.pasteur.fr/bli/qaf_demux/blob/master/Julia/QafDemux/bin/snoop.jl#L8) you pass in a `Vector{SubString}` to `main`.

But I see the `ERROR: Unable to find compatible target in system image.` now which I guess is the real problem.

---

<div class="post-metadata">

**Author:** ![blaiseli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/blaiseli/32/10332_2.png) [@blaiseli](https://discourse.julialang.org/u/blaiseli)\
**Post date:** [October 2, 2019, 5:24pm UTC](https://discourse.julialang.org/t/determining-whether-julia-is-self-built-or-shipped/29404/7 "2019-10-02T17:24:18Z")

</div>

> [@kristoffer.carlsson](#):
>
> What if you just fix `make_record_reader` to take an `AbstractString` as the first argument?

I then get another error: `LoadError: MethodError: no method matching chosetranscoder(::SubString{String})`

This too can probably fixed by changing type signatures of my functions, but how come this `cpu-target` compiling option interacts with allowed type signatures in my code?