Creating a system image on Raspery Pi 5

General Usage
,
1

I wrote a script to create a system image on the Pi 5 for my current project, see:

I am using ModelingToolkit and other libraries that result in a very large system image.

This causes some issues, mainly the RAM requirements are huge.
When using one thread I need at least 24 GB RAM, for multi-threaded system image creation more. This is an issue if you have only 8GB physical RAM. Solution:

  1. use a swap file of at least 16 GB
  2. use and enable zswap, see: zswap - ArchWiki
  3. turn off systemd-oomd, because on Ubuntu 23.10 which I am using it would
    otherwise kill the Julia compilation process even if there is enougth swap
    available

Now the script succeeds in about 52 minutes. On my desktop, based on a 16 core AMD CPU with 32 GB RAM it needs only 7:22 minutes (tested after deleting the .julia folder).

Here the full script, not yet tested on Windows:

#!/bin/bash -eu
if [[ $(basename $(pwd)) == "bin" ]]; then
    cd ..
fi

if ! command -v juliaup &> /dev/null; then
    echo "Please install the Julia installer 'juliaup'!"
    echo "See: https://github.com/JuliaLang/juliaup"
    exit 1
fi

juliaup add 1.10
juliaup default 1.10

export KMP_DUPLICATE_LIB_OK=TRUE
export MPLBACKEND=qt5agg
export PYTHON=""

# total memory in kB
MEM=$(grep MemTotal /proc/meminfo | awk '{print $2}')

if (( $MEM <  15000000 )); then
    sudo /bin/bash -c "echo '1' | tee -a /proc/sys/vm/overcommit_memory"
    sudo systemctl disable --now systemd-oomd
    export JULIA_IMAGE_THREADS=1
    export JULIA_PKG_PRESERVE_TIERED_INSTALLED=true
fi

julia_version=$(julia --version | awk '{print($3)}')
julia_major=${julia_version:0:3} 
if [[ $julia_major == "1.1" ]]; then
    julia_major=${julia_version:0:4} 
fi
if test -f "kps-image-${julia_major}.so"; then
    mv bin/kps-image-${julia_major}.so bin/kps-image-${julia_major}.so.bak
fi

echo "Updating packages..."
if test -f "Manifest.toml"; then
   rm Manifest.toml
fi
julia --pkgimages=no --project -e "using Pkg; Pkg.add(\"PyCall\"); Pkg.build(\"PyCall\")"
julia --pkgimages=no --project -e "using Pkg; Pkg.add(\"Conda\"); using Conda; Conda.add(\"matplotlib\")"
julia --pkgimages=no --project -e "using PyPlot"
julia --pkgimages=no --project -e "include(\"./test/update_packages.jl\");"

julia --pkgimages=no --project -t 4 -e "using Pkg; Pkg.precompile()"
if [[ $julia_major == "1.9" ]]; then
    julia --pkgimages=no --project -t 4 -e "include(\"./test/create_sys_image.jl\");"
else
    julia --pkgimages=no --project -t 4 --gcthreads=4,1 -e "include(\"./test/create_sys_image.jl\");"
fi
mv kps-image_tmp.so bin/kps-image-${julia_major}.so

Questions:

  • why does compiling ModelingToolkit.jl needs so much memory?
  • any other idea how to bring this time down to < 30 min on a Pi 5?

When the image is compiled everything runs fast and smooth on the Pi.

4 Likes
2

One idea would be to use another ARM system and then set JULIA_CPU_TARGET to target the Raspberry Pi.

Try the following for options.

julia -C help

As for why it is such a resource hog, this is mainly the domain of LLVM I think.

You could also use PkgCacheInspector.jl to investigate the pkgimage of ModelingToolkit.jl.

3

Not using qt5agg saves 5 minutes:

#!/bin/bash -eu
if [[ $(basename $(pwd)) == "bin" ]]; then
    cd ..
fi

if ! command -v juliaup &> /dev/null; then
    echo "Please install the Julia installer 'juliaup'!"
    echo "See: https://github.com/JuliaLang/juliaup"
    exit 1
fi

juliaup add 1.10
juliaup default 1.10

export KMP_DUPLICATE_LIB_OK=TRUE
export PYTHON=""

# total memory in kB
MEM=$(grep MemTotal /proc/meminfo | awk '{print $2}')

if (( $MEM <  15000000 )); then
    sudo /bin/bash -c "echo '1' | tee -a /proc/sys/vm/overcommit_memory"
    sudo systemctl disable --now systemd-oomd
    export JULIA_IMAGE_THREADS=1
    export JULIA_PKG_PRESERVE_TIERED_INSTALLED=true
else
    export MPLBACKEND=qt5agg
fi

julia_version=$(julia --version | awk '{print($3)}')
julia_major=${julia_version:0:3} 
if [[ $julia_major == "1.1" ]]; then
    julia_major=${julia_version:0:4} 
fi
if test -f "kps-image-${julia_major}.so"; then
    mv bin/kps-image-${julia_major}.so bin/kps-image-${julia_major}.so.bak
fi

echo "Updating packages..."
if test -f "Manifest.toml"; then
   rm Manifest.toml
fi
julia --pkgimages=no --project -e "using Pkg; Pkg.add(\"PyCall\"); Pkg.build(\"PyCall\")"
julia --pkgimages=no --project -e "using Pkg; Pkg.add(\"Conda\"); using Conda; Conda.add(\"matplotlib\")"
julia --pkgimages=no --project -e "using PyPlot"
julia --pkgimages=no --project -e "include(\"./test/update_packages.jl\");"

julia --pkgimages=no --project -t auto -e "using Pkg; Pkg.precompile()"
julia --pkgimages=no --project -t auto --gcthreads=2,1 -e "include(\"./test/create_sys_image.jl\");"

mv kps-image_tmp.so bin/kps-image-${julia_major}.so

This needs:

✔ [31m:42s] PackageCompiler: compiling incremental system image

real	47m11.794s
user	42m14.517s
sys	    13m44.359s