Secure systems: an alternate use case for PackageCompiler.jl

General Usage
1

Julia’s package manager relies on internet access, so it can be challenging to make dependencies available on secure systems. PackageCompiler.jl compiles Julia code ahead of time, primarily for improved performance in certain workflows. In particular, generating a sysimage produces a performant stand-alone Julia session, and since its main drawback (version locking) is mitigated on secure systems, it seems to be the best approach for putting a Julia session on a system without internet access or direct ssh access.

@dilumaluthge and I have put together a workflow for uploading Julia sessions to our institution’s secure system, and we wanted to make a public version available. SIEGE takes a Julia project as input and automatically generates a sysimage (using PackageCompiler) for use in secure systems. It also includes a Bash script that starts an offline Julia session with the new sysimage. The script sets environment variables and accepts command-line arguments, so it can be aliased to julia by users. The main requirement is an internet-connected build server that has the same architecture as the secure system.

I don’t believe SIEGE is the very first tool for automatically generating sysimages, but I couldn’t find much documentation on using sysimages for secure systems, so sharing our approach should hopefully save other researchers some time and effort.

14 Likes
2

The repo gives more details. Broadly speaking, we used the following approach:

  1. Build a sysimage with PackageCompiler
  2. Upload the build and Julia binaries to the secure system
  3. Alias a script to julia, similar to the following example:
#!/bin/bash
set -Eeu -o pipefail

unset JULIA_LOAD_PATH

export JULIA_PROJECT=/Path/To/Project
export JULIA_DEPOT_PATH=/Path/To/Depot
export JULIA_PKG_OFFLINE=true

/Path/To/julia -J/Path/To/sysimage.so "$@"
3 Likes
3

Hello, I’m so excited to see a solution for Julia in secure system. I’ve given it a try but something went wrong here, so I guess I must have missed something.

Here is my procedure:

  1. git clone SIEGE, and follow the instructions in github README.
  2. before running main(“path_to_SIEGE”), I typed “using LoopVectorization”, and did some unrelevant calculation with @turbo.
  3. Then I ran main(“path_to_SIEGE”). After that, I can see a “sysimage.so” in SIEGE/build folder.
  4. Then I upload the whole SIEGE folder to the secure system, and edited run_sysimage.sh to fit my environment (actually what I did is just completing the path).
  5. Then I ran “run_sysimage.sh”, and julia was successfully executed.

Now here is my problem: I thought that now that I wrapped all the running environment into a .so, and I started another julia from that .so, I should natrually have the package “LoopVectorization”. But the fact is that I can’t “using LoopVectorization”, nor use @turbo.

So in a nutshell, what should I do to transfer all my required packages to that secure system? Do I have to edit Manifest.toml or something?

Thanks in advance!

4

SIEGE builds the sysimage and artifacts based on env/Project.toml, so you would need to add the package to that environment.

1 Like
5

So much thanks! It did work.