# Using PackageCompiler to create a custom system image
**URL:** https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528
**Category:** General Usage
**Created:** [May 23, 2019, 4:32pm UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528 "2019-05-23T16:32:35Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)
#### Post date: [May 23, 2019, 4:32pm UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/1 "2019-05-23T16:32:35Z")
Hi guys, I need help to create a custom system image Using PackageCompiler.
I need to build a system image with a package bundled. The idea is to send this system image to another person that will be able to use my package without downloading anything.
I could create a custom system image using `compile_package` function, but when I use this image in another computer, it does not locate the package.
I was able to do this in the past using the `build_sysimg.jl` script.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [May 23, 2019, 5:07pm UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/2 "2019-05-23T17:07:17Z")
Have you thought of using containers for this purpose? Docker or Singularity?
---
### Author:  [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)
#### Post date: [May 23, 2019, 5:08pm UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/3 "2019-05-23T17:08:01Z")
The purpose is to embed this Julia image inside a C++ software. It must run on different platforms. Hence, containers are not an option here.
---
### Author:  [@Pbellive](https://discourse.julialang.org/u/Pbellive)
#### Post date: [May 23, 2019, 6:08pm UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/4 "2019-05-23T18:08:35Z")
Will the other computer you’re sending the sysimage to have julia installed? I’m guessing yes since you’re building a sysimage and not just a shared library that you’ll be linking your C++ executable against.
Anyway, assuming the target computer has julia installed I’ve gotten this to work with the `build_sysimg` function from the current PackageCompiler. The trick is that the package compiled into the system image needs to be listed in the Project.toml file of the active environment on the target machine. The source code doesn’t need to exist on the target machine but you do seem to need the package listed in Project.toml. I haven’t found a way around that requirement. A big limitation of this approach is that it breaks the package manager on the target machine. So if you have a package `Foo` that is compiled into the sys image and listed in Project.toml but the source code for `Foo` isn’t in the depot on the target machine then `using Foo` will work as expected on the target machine but any package manager commands (e.g. `]add Bar` for some unrelated package Bar will error.
---
### Author:  [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)
#### Post date: [May 23, 2019, 7:18pm UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/5 "2019-05-23T19:18:11Z")
The missing `Project.toml` explains a lot (why it works on the source computer and not on the destination)! I wonder if I can add this package inside the `stdlib` so that it works as any other in there.
---
### Author:  [@Pbellive](https://discourse.julialang.org/u/Pbellive)
#### Post date: [May 23, 2019, 7:32pm UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/6 "2019-05-23T19:32:42Z")
I looked briefly at hacking my code into stdlib to avoid the Project.toml requirement. I didn’t get it to work but it might be possible. Let me know if you do get it to work.
---
### Author:  [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)
#### Post date: [May 24, 2019, 12:07am UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/7 "2019-05-24T00:07:05Z")
In the mean time, I managed to get a (very very very) dirty workaround: recompile the sysimg by hand 😃
1. Go to `share/julia/base/` and copy the folder of the package. In my case it is named `ForplanSimulatorCore`.
2. Run Julia and make sure all dependencies of the package are installed (I just activate it and then run `instantiate`).
3. Inside the directory `base`, create a `ushering.jl` with:
4. Run:
5. Now, in `../tmp` you have `sys.ji` and `sys.o`. You can run `julia -J sys.ji`, but it will be painfully slow. However, you can use it to test.
6. Finally, we have to link. The command depends on the architecture. I figured out by looking in the `PackageCompiler` source code. For macOS, it should be more or less:
Then, you can run `julia -J sys.dylib`. The package is then loaded by:
```julia
julia> using .ForplanSimulatorCore
```
Everything seems to be ok 🙂
**EDIT** : It turns out that you do not need to copy the package into `base`. Just add the packages using `Pkg`, **make sure all of them are precompiled** , and add only
```julia
using ForplanSimulatorCore
```
on `userimg.jl`.
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [May 24, 2019, 1:00am UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/8 "2019-05-24T01:00:33Z")
> [@Ronis\_BR](#):
>
> The idea is to send this system image to another person that will be able to use my package without downloading anything.
I think this approach would not work if the Julia packages you want to use contain some external resources. For example, packages `ccall`s external libraries often store full path to the libraries. Some Julia packages need non-Julia file assets (e.g., HTML files). I don’t think the system image is relocatable yet in such cases. But you don’t need to worry about this if you are only including pure-Julia code that does not need any assets.
See also: [Pkg + BinaryProvider · Issue #841 · JuliaLang/Pkg.jl · GitHub](https://github.com/JuliaLang/Pkg.jl/issues/841#issuecomment-470819934)
---
### Author:  [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)
#### Post date: [May 24, 2019, 1:18am UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/9 "2019-05-24T01:18:31Z")
Thank you very much to point this! Fortunately, my code is pure Julia but I could make this mistake sometime 🙂
**EDIT** : @tkf just for curiosity, is there any plans to fix such things?
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [May 24, 2019, 2:15am UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/10 "2019-05-24T02:15:10Z")
For external libraries, there is [@staticfloat’s response](https://github.com/JuliaLang/Pkg.jl/issues/841#issuecomment-475370178) to [@phlavenk’s comment](https://github.com/JuliaLang/Pkg.jl/issues/841#issuecomment-470819934) (which I just linked above). But it looks like this approach would require shipping the external libraries together with the system image. Just mentioning this as you may want single-file approach. Anyway, it looks to me that it is solvable by improvements in BinaryProvider (if the packages are using BinaryProvider; other packages, e.g., PyCall, need their own fix). I think the best way to track the improvements is to ask BinaryProvider devs.
For other kinds assets like HTML files, I don’t know if there is a simple ready-to-use solution. But I think this is relatively easy to solve. For example, you can convert assets to `repr` of Julia objects and dump them into `.jl` files in the build step.
---
### Author:  [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)
#### Post date: [May 24, 2019, 2:17am UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/11 "2019-05-24T02:17:36Z")
> [@tkf](#):
>
> For other kinds assets like HTML files, I don’t know if there is a simple ready-to-use solution. But I think this is relatively easy to solve. For example, you can convert assets to `repr` of Julia objects and dump them into `.jl` files in the build step.
Thanks! This is a very nice advice. In my case, the only things bundled in the package (which I do not plan to use in a short time frame) are the coefficients of three gravity models. I think the `repr` approach can help me 🙂
---
### Author:  [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)
#### Post date: [May 24, 2019, 2:20am UTC](https://discourse.julialang.org/t/using-packagecompiler-to-create-a-custom-system-image/24528/12 "2019-05-24T02:20:14Z")
If anyone wants, here is the script I am using to create a system image bundling three packages: SatelliteToolbox.jl, ReferenceFrameRotations.jl, and ForplanSimulatorCore.jl (this is private).
```julia
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
#
# Description
#
# This script is used to compile the Julia system image of the Forplan
# satellite simulator.
#
# This script was based on the code of PackageCompiler.jl.
#
# Note: This will only work in Linux or macOS!
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
using Pkg
using Libdl
function verify_compiler(cc)
try
return success(`$cc --version`)
catch
return false
end
end
function get_c_compiler()
# The environment variable CC has the precedence.
if haskey(ENV, "CC")
cc = "$(ENV["CC"])"
if !verify_compiler(cc)
@error("The compiler in environment variable CC = `$cc` does not seem to work.")
else
return cc
end
end
# Check for compilers.
candidates_cc = ["clang", "gcc"]
for cc in candidates_cc
if verify_compiler(cc)
return cc
end
end
return nothing
end
# Auxiliary variables
# ==============================================================================
base_dir = dirname(Base.find_source_file("sysimg.jl"))
lib_dir = dirname(Libdl.dlpath("libjulia"))
cc = get_c_compiler()
julia = Base.julia_cmd()[1]
# Check if a C compiler was found.
if cc == nothing
error("No suitable C compiler was found!")
exit()
end
# Get the linking flags with respect to the OS.
cflags = ["-shared", "-std=gnu99", "-O3"]
ldflags = ["-L$lib_dir"]
ldlibs = ["-ljulia"]
if Sys.isunix()
push!(cflags, "-fPIC")
end
if Sys.islinux()
push!(ldflags, "-Wl,--export-dynamic")
end
# Install and pre-compile the dependencies
# ==============================================================================
@info("Installing and pre-compiling the dependencies.")
Pkg.add("ReferenceFrameRotations")
Pkg.add("SatelliteToolbox")
Pkg.add(PackageSpec(path="../SimulatorCore/ForplanSimulatorCore.jl"))
using ReferenceFrameRotations
using SatelliteToolbox
using ForplanSimulatorCore
println()
println()
# Create the system image
# ==============================================================================
sysimg_path = pwd() * "/tmp"
try
# The files will be created in `./tmp` directory.
rm("./tmp", recursive = true, force = true)
mkdir("./tmp")
cd("./tmp")
cd(base_dir)
# Abort if there are an `userimg.jl` file.
if isfile("userimg.jl") || isdir("userimg.jl")
error("userimg.jl exists. Aborting...")
end
# Create the `userimg.jl` file.
open("userimg.jl","w") do f
write(f, "using ForplanSimulatorCore\n")
write(f, "using ReferenceFrameRotations\n")
write(f, "using SatelliteToolbox\n")
end
# Build the base compiler.
@info("Building the base compiler.")
run(`$julia -C core2
--output-ji $sysimg_path/basecompiler.ji
--output-o $sysimg_path/basecompiler.o
compiler/compiler.jl`)
println()
println()
@info("Building the system image.")
run(`$julia -C core2
--output-ji $sysimg_path/sys.ji
--output-o $sysimg_path/sys.o
-J $sysimg_path/basecompiler.ji
--startup-file=no sysimg.jl`)
println()
println()
@info("Linking the system image.")
cd(sysimg_path)
syso = Sys.isapple() ? `-Wl,-all_load sys.o` :
`-Wl,--whole-archive sys.o -Wl,--no-whole-archive`
sysf = "sys." * Libdl.dlext
run(`$cc $cflags $ldflags -o $sysf $syso $ldlibs`)
cp("$sysf", "../$sysf")
println()
println()
@info("Removing the package from the environment.")
Pkg.rm("ForplanSimulatorCore")
println()
println()
catch e
showerror(stderr,e,catch_backtrace())
println()
println()
finally
# Delete the temporary files, even an error occurred.
@info("Deleting temporary files.")
rm(base_dir * "/userimg.jl"; force = true)
rm(sysimg_path, recursive = true, force = true)
end
```