Hi guys, I need help to create a custom system image Using PackageCompiler.
I need to build a system image with a package bundled. The idea is to send this system image to another person that will be able to use my package without downloading anything.
I could create a custom system image using compile_package function, but when I use this image in another computer, it does not locate the package.
I was able to do this in the past using the build_sysimg.jl script.
Have you thought of using containers for this purpose? Docker or Singularity?
The purpose is to embed this Julia image inside a C++ software. It must run on different platforms. Hence, containers are not an option here.
Will the other computer you’re sending the sysimage to have julia installed? I’m guessing yes since you’re building a sysimage and not just a shared library that you’ll be linking your C++ executable against.
Anyway, assuming the target computer has julia installed I’ve gotten this to work with the build_sysimg function from the current PackageCompiler. The trick is that the package compiled into the system image needs to be listed in the Project.toml file of the active environment on the target machine. The source code doesn’t need to exist on the target machine but you do seem to need the package listed in Project.toml. I haven’t found a way around that requirement. A big limitation of this approach is that it breaks the package manager on the target machine. So if you have a package Foo that is compiled into the sys image and listed in Project.toml but the source code for Foo isn’t in the depot on the target machine then using Foo will work as expected on the target machine but any package manager commands (e.g. ]add Bar for some unrelated package Bar will error.
The missing Project.toml explains a lot (why it works on the source computer and not on the destination)! I wonder if I can add this package inside the stdlib so that it works as any other in there.
I looked briefly at hacking my code into stdlib to avoid the Project.toml requirement. I didn’t get it to work but it might be possible. Let me know if you do get it to work.
In the mean time, I managed to get a (very very very) dirty workaround: recompile the sysimg by hand 
Go to share/julia/base/ and copy the folder of the package. In my case it is named ForplanSimulatorCore.
Run Julia and make sure all dependencies of the package are installed (I just activate it and then run instantiate).
Inside the directory base, create a ushering.jl with:
push!(LOAD_PATH, "./ForplanSimulatorCore")
using ForplanSimulatorCore
Run:
julia -C core2 --output-ji ../tmp/basecompiler.ji --output-o ../tmp/basecompiler.o compiler/compiler.jl
julia -C core2 --output-ji ../tmp/sys.ji --output-o ../tmp/sys.o -J ../tmp/basecompiler.ji --startup-file=no sysimg.jl
Change ../tmp to the desired directory.
Now, in ../tmp you have sys.ji and sys.o. You can run julia -J sys.ji, but it will be painfully slow. However, you can use it to test.
Finally, we have to link. The command depends on the architecture. I figured out by looking in the PackageCompiler source code. For macOS, it should be more or less:
g++ --shared -std=gnu99 -fPIC -L/Applications/Julia-1.1.app/Contents/Resources/julia/lib -o sys.dylib -Wl,-all_load sys.o -O3 -ljulia
Then, you can run julia -J sys.dylib. The package is then loaded by:
julia> using .ForplanSimulatorCore
Everything seems to be ok 
EDIT: It turns out that you do not need to copy the package into base. Just add the packages using Pkg, make sure all of them are precompiled, and add only
using ForplanSimulatorCore
on userimg.jl.
I think this approach would not work if the Julia packages you want to use contain some external resources. For example, packages ccalls external libraries often store full path to the libraries. Some Julia packages need non-Julia file assets (e.g., HTML files). I don’t think the system image is relocatable yet in such cases. But you don’t need to worry about this if you are only including pure-Julia code that does not need any assets.
See also: Pkg + BinaryProvider · Issue #841 · JuliaLang/Pkg.jl · GitHub
Thank you very much to point this! Fortunately, my code is pure Julia but I could make this mistake sometime
EDIT: @tkf just for curiosity, is there any plans to fix such things?
For external libraries, there is @staticfloat’s response to @phlavenk’s comment (which I just linked above). But it looks like this approach would require shipping the external libraries together with the system image. Just mentioning this as you may want single-file approach. Anyway, it looks to me that it is solvable by improvements in BinaryProvider (if the packages are using BinaryProvider; other packages, e.g., PyCall, need their own fix). I think the best way to track the improvements is to ask BinaryProvider devs.
For other kinds assets like HTML files, I don’t know if there is a simple ready-to-use solution. But I think this is relatively easy to solve. For example, you can convert assets to repr of Julia objects and dump them into .jl files in the build step.
Thanks! This is a very nice advice. In my case, the only things bundled in the package (which I do not plan to use in a short time frame) are the coefficients of three gravity models. I think the repr approach can help me
If anyone wants, here is the script I am using to create a system image bundling three packages: SatelliteToolbox.jl, ReferenceFrameRotations.jl, and ForplanSimulatorCore.jl (this is private).
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
#
# Description
#
# This script is used to compile the Julia system image of the Forplan
# satellite simulator.
#
# This script was based on the code of PackageCompiler.jl.
#
# Note: This will only work in Linux or macOS!
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
using Pkg
using Libdl
function verify_compiler(cc)
try
return success(`$cc --version`)
catch
return false
end
end
function get_c_compiler()
# The environment variable CC has the precedence.
if haskey(ENV, "CC")
cc = "$(ENV["CC"])"
if !verify_compiler(cc)
@error("The compiler in environment variable CC = `$cc` does not seem to work.")
else
return cc
end
end
# Check for compilers.
candidates_cc = ["clang", "gcc"]
for cc in candidates_cc
if verify_compiler(cc)
return cc
end
end
return nothing
end
# Auxiliary variables
# ==============================================================================
base_dir = dirname(Base.find_source_file("sysimg.jl"))
lib_dir = dirname(Libdl.dlpath("libjulia"))
cc = get_c_compiler()
julia = Base.julia_cmd()[1]
# Check if a C compiler was found.
if cc == nothing
error("No suitable C compiler was found!")
exit()
end
# Get the linking flags with respect to the OS.
cflags = ["-shared", "-std=gnu99", "-O3"]
ldflags = ["-L$lib_dir"]
ldlibs = ["-ljulia"]
if Sys.isunix()
push!(cflags, "-fPIC")
end
if Sys.islinux()
push!(ldflags, "-Wl,--export-dynamic")
end
# Install and pre-compile the dependencies
# ==============================================================================
@info("Installing and pre-compiling the dependencies.")
Pkg.add("ReferenceFrameRotations")
Pkg.add("SatelliteToolbox")
Pkg.add(PackageSpec(path="../SimulatorCore/ForplanSimulatorCore.jl"))
using ReferenceFrameRotations
using SatelliteToolbox
using ForplanSimulatorCore
println()
println()
# Create the system image
# ==============================================================================
sysimg_path = pwd() * "/tmp"
try
# The files will be created in `./tmp` directory.
rm("./tmp", recursive = true, force = true)
mkdir("./tmp")
cd("./tmp")
cd(base_dir)
# Abort if there are an `userimg.jl` file.
if isfile("userimg.jl") || isdir("userimg.jl")
error("userimg.jl exists. Aborting...")
end
# Create the `userimg.jl` file.
open("userimg.jl","w") do f
write(f, "using ForplanSimulatorCore\n")
write(f, "using ReferenceFrameRotations\n")
write(f, "using SatelliteToolbox\n")
end
# Build the base compiler.
@info("Building the base compiler.")
run(`$julia -C core2
--output-ji $sysimg_path/basecompiler.ji
--output-o $sysimg_path/basecompiler.o
compiler/compiler.jl`)
println()
println()
@info("Building the system image.")
run(`$julia -C core2
--output-ji $sysimg_path/sys.ji
--output-o $sysimg_path/sys.o
-J $sysimg_path/basecompiler.ji
--startup-file=no sysimg.jl`)
println()
println()
@info("Linking the system image.")
cd(sysimg_path)
syso = Sys.isapple() ? `-Wl,-all_load sys.o` :
`-Wl,--whole-archive sys.o -Wl,--no-whole-archive`
sysf = "sys." * Libdl.dlext
run(`$cc $cflags $ldflags -o $sysf $syso $ldlibs`)
cp("$sysf", "../$sysf")
println()
println()
@info("Removing the package from the environment.")
Pkg.rm("ForplanSimulatorCore")
println()
println()
catch e
showerror(stderr,e,catch_backtrace())
println()
println()
finally
# Delete the temporary files, even an error occurred.
@info("Deleting temporary files.")
rm(base_dir * "/userimg.jl"; force = true)
rm(sysimg_path, recursive = true, force = true)
end