PackageCompiler crash after 6 hours of compilation, seemingly running out of memory

ErikBuer · June 16, 2024, 8:45am

Hi.

I am attempting to build an application with create_app that will run on a Raspberry Pi 5 (8 GB RAM). I am therefore also compiling my application on the PI. The PI is running the latest Rasbian and is a 64-bit device. I’m running on a 16 GB SD card for the OS, but is outputting the application to a memory stick with lots of storage space.

The code is not massive, but it takes forever to compile, and has thus far, crashed every time.

julia> versioninfo()
Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (aarch64-linux-gnu)
  CPU: 4 × Cortex-A76
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, cortex-a76)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
  JULIA_CONDAPKG_BACKEND = Null
  JULIA_PYTHONCALL_EXE = /usr/bin/python3
  JULIA_NUM_THREADS = 4

julia> create_app(".", "/home/myUser/storage/MyApp", incremental=true, force=true)
PackageCompiler: bundled libraries:
  ├── Base:
  │    ├── libLLVM-15jl.so - 80.875 MiB
  │    ├── libatomic.so.1.2.0 - 241.727 KiB
  │    ├── libdSFMT.so - 65.727 KiB
  │    ├── libgcc_s.so.1 - 582.016 KiB
  │    ├── libgfortran.so.5.0.0 - 6.107 MiB
  │    ├── libgmp.so.10.4.1 - 475.820 KiB
  │    ├── libgmpxx.so.4.6.1 - 38.641 KiB
  │    ├── libgomp.so.1.0.0 - 1.435 MiB
  │    ├── libjulia-codegen.so.1.10.4 - 64.452 MiB
  │    ├── libjulia-internal.so.1.10.4 - 12.554 MiB
  │    ├── libmpfr.so.6.2.0 - 2.515 MiB
  │    ├── libopenlibm.so.4.0 - 430.963 KiB
  │    ├── libpcre2-8.so.0.11.2 - 567.703 KiB
  │    ├── libssp.so.0.0.0 - 89.047 KiB
  │    ├── libstdc++.so.6.0.32 - 19.951 MiB
  │    ├── libunwind.so.8.0.1 - 579.992 KiB
  │    ├── libuv.so.2.0.0 - 718.633 KiB
  │    ├── libz.so.1.2.13 - 106.587 KiB
  │    ├── libjulia.so.1.10.4 - 265.258 KiB
  ├── Stdlibs:
  │   ├── LibGit2_jll
  │   │   ├── libgit2.so.1.6.4 - 1.403 MiB
  │   ├── SuiteSparse_jll
  │   │   ├── libamd.so.3.2.1 - 65.516 KiB
  │   │   ├── libbtf.so.2.2.1 - 65.117 KiB
  │   │   ├── libcamd.so.3.2.1 - 65.555 KiB
  │   │   ├── libccolamd.so.3.2.1 - 65.586 KiB
  │   │   ├── libcholmod.so.4.2.1 - 1.159 MiB
  │   │   ├── libcolamd.so.3.2.1 - 65.289 KiB
  │   │   ├── libklu.so.2.2.1 - 195.539 KiB
  │   │   ├── libldl.so.3.2.1 - 65.383 KiB
  │   │   ├── librbio.so.4.2.1 - 65.453 KiB
  │   │   ├── libspqr.so.4.2.1 - 415.094 KiB
  │   │   ├── libsuitesparseconfig.so.7.2.1 - 14.149 KiB
  │   │   ├── libumfpack.so.6.2.1 - 716.008 KiB
  │   ├── libblastrampoline_jll
  │   │   ├── libblastrampoline.so.5 - 2.673 MiB
  │   ├── OpenBLAS_jll
  │   │   ├── libopenblas64_.0.3.23.so - 22.221 MiB
  │   ├── LibSSH2_jll
  │   │   ├── libssh2.so.1.0.1 - 327.594 KiB
  │   ├── LibCURL_jll
  │   │   ├── libcurl.so.4.8.0 - 650.422 KiB
  │   ├── MbedTLS_jll
  │   │   ├── libmbedcrypto.so.2.28.2 - 645.416 KiB
  │   │   ├── libmbedtls.so.2.28.2 - 267.172 KiB
  │   │   ├── libmbedx509.so.2.28.2 - 202.734 KiB
  │   ├── nghttp2_jll
  │   │   ├── libnghttp2.so.14.24.1 - 830.447 KiB
  Total library file size: 224.022 MiB
PackageCompiler: bundled artifacts:
  ├── FFTW_jll - 2.725 MiB
  ├── HDF5_jll - 12.965 MiB
  ├── Hwloc_jll - 6.790 MiB
  ├── MPICH_jll - 10.584 MiB
  ├── OpenSSL_jll - 8.598 MiB
  ├── OpenSpecFun_jll - 487.436 KiB
  ├── ZeroMQ_jll - 884.858 KiB
  ├── libaec_jll - 211.364 KiB
  ├── libsodium_jll - 4.119 MiB
  └── oneTBB_jll - 1.713 MiB
  Total artifact file size: 49.039 MiB
Precompiling project...
  69 dependencies successfully precompiled in 91 seconds. 6 already precompiled.
⢰ [06h:45m:35s] PackageCompiler: compiling incremental system image
┌ Warning: Free system memory dropped to 42.031 MiB during sysimage compilation.
│ If the reason the subprocess errored isn't clear, it may have been OOM-killed.
└ @ PackageCompiler ~/.julia/packages/PackageCompiler/nT5sD/src/PackageCompiler.jl:137
ERROR: failed process: Process(`/home/myUser/.julia/juliaup/julia-1.10.4+0.aarch64.linux.gnu/bin/julia --color=yes --startup-file=no --pkgimages=no --cpu-target=generic --sysimage=/home/myUser/.julia/juliaup/julia-1.10.4+0.aarch64.linux.gnu/lib/julia/sys.so --project=/home/myUser/myRepo/julia --output-o=/tmp/jl_7qTwNyT4Be.o /tmp/jl_9wxCNEbEtE`, ProcessSignaled(9)) [0]

When I run with incremental=false, it gives the same warning, but after 45 minutes.

I have access to a lot more powerful hardware, but it seems that cross-compilation is not possible with PackageCompiler.

Do you have any recommendations on what to try next, or maybe a solution for cross-compilation?

Thanks!

ufechner7 · June 16, 2024, 1:17pm

zram might help, see: Building Julia on a Raspberry Pi - #5 by Oscar_Smith

And: examples/how_to_enable_ZRAM_Raspberry_Pi.md at master · garyexplains/examples · GitHub

And a larger swap file: How to Change Swap File Size in Ubuntu – TecAdmin

And perhaps disable the option to build the image using multiple threads.

I was able to tune my system to be able to build my system image within 45 minutes… Still a long time. Next step would be to use a pci SSD to store the swap file to see if that can speed things up.

ErikBuer · July 2, 2024, 6:53pm

Thanks for the tips @ufechner7

I’ll try to make a dedicated development pi (build-machine) with an SSD.
That way I can also increase the SWAP, as you suggested

I wonder how much improvement I would see running an SSD with a DRAM cache.

ErikBuer · July 7, 2024, 2:08pm

I migrated the sd-card image to an SSD and increased the SWAP file to 10 GB.
Running with 8 GB of RAM, I now managed to compile in about 10-15 minutes.

I used a cheap 250 GB SSD without DRAM cache.
I did change the PCIe speed on the pi to PCIe gen. 3 (more info here).

julia> create_app(".", "/home/someFolder", incremental=false, force=true, include_lazy_artifacts=true)
PackageCompiler: bundled libraries:
  ├── Base:
  │    ├── libLLVM-15jl.so - 80.875 MiB
  │    ├── libatomic.so.1.2.0 - 241.727 KiB
  │    ├── libdSFMT.so - 65.727 KiB
  │    ├── libgcc_s.so.1 - 582.016 KiB
  │    ├── libgfortran.so.5.0.0 - 6.107 MiB
  │    ├── libgmp.so.10.4.1 - 475.820 KiB
  │    ├── libgmpxx.so.4.6.1 - 38.641 KiB
  │    ├── libgomp.so.1.0.0 - 1.435 MiB
  │    ├── libjulia-codegen.so.1.10.4 - 64.452 MiB
  │    ├── libjulia-internal.so.1.10.4 - 12.554 MiB
  │    ├── libmpfr.so.6.2.0 - 2.515 MiB
  │    ├── libopenlibm.so.4.0 - 430.963 KiB
  │    ├── libpcre2-8.so.0.11.2 - 567.703 KiB
  │    ├── libssp.so.0.0.0 - 89.047 KiB
  │    ├── libstdc++.so.6.0.32 - 19.951 MiB
  │    ├── libunwind.so.8.0.1 - 579.992 KiB
  │    ├── libuv.so.2.0.0 - 718.633 KiB
  │    ├── libz.so.1.2.13 - 106.587 KiB
  │    ├── libjulia.so.1.10.4 - 265.258 KiB
  ├── Stdlibs:
  │   ├── LibGit2_jll
  │   │   ├── libgit2.so.1.6.4 - 1.403 MiB
  │   ├── SuiteSparse_jll
  │   │   ├── libamd.so.3.2.1 - 65.516 KiB
  │   │   ├── libbtf.so.2.2.1 - 65.117 KiB
  │   │   ├── libcamd.so.3.2.1 - 65.555 KiB
  │   │   ├── libccolamd.so.3.2.1 - 65.586 KiB
  │   │   ├── libcholmod.so.4.2.1 - 1.159 MiB
  │   │   ├── libcolamd.so.3.2.1 - 65.289 KiB
  │   │   ├── libklu.so.2.2.1 - 195.539 KiB
  │   │   ├── libldl.so.3.2.1 - 65.383 KiB
  │   │   ├── librbio.so.4.2.1 - 65.453 KiB
  │   │   ├── libspqr.so.4.2.1 - 415.094 KiB
  │   │   ├── libsuitesparseconfig.so.7.2.1 - 14.149 KiB
  │   │   ├── libumfpack.so.6.2.1 - 716.008 KiB
  │   ├── libblastrampoline_jll
  │   │   ├── libblastrampoline.so.5 - 2.673 MiB
  │   ├── OpenBLAS_jll
  │   │   ├── libopenblas64_.0.3.23.so - 22.221 MiB
  │   ├── LibSSH2_jll
  │   │   ├── libssh2.so.1.0.1 - 327.594 KiB
  │   ├── LibCURL_jll
  │   │   ├── libcurl.so.4.8.0 - 650.422 KiB
  │   ├── MbedTLS_jll
  │   │   ├── libmbedcrypto.so.2.28.2 - 645.416 KiB
  │   │   ├── libmbedtls.so.2.28.2 - 267.172 KiB
  │   │   ├── libmbedx509.so.2.28.2 - 202.734 KiB
  │   ├── nghttp2_jll
  │   │   ├── libnghttp2.so.14.24.1 - 830.447 KiB
  Total library file size: 224.022 MiB
PackageCompiler: bundled artifacts:
  ├── FFTW_jll - 2.725 MiB
  ├── HDF5_jll - 12.965 MiB
  ├── Hwloc_jll - 6.790 MiB
  ├── MPICH_jll - 10.584 MiB
  ├── OpenSSL_jll - 8.592 MiB
  ├── OpenSpecFun_jll - 487.436 KiB
  ├── ZeroMQ_jll - 884.858 KiB
  ├── libaec_jll - 211.364 KiB
  ├── libsodium_jll - 4.119 MiB
  ├── micromamba_jll - 16.830 MiB
  └── oneTBB_jll - 1.713 MiB
  Total artifact file size: 65.864 MiB
✔ [03m:17s] PackageCompiler: creating compiler .ji image (incremental=false)
✔ [04m:06s] PackageCompiler: compiling fresh sysimage (incremental=false)
┌ Warning: The active manifest file is an older format with no julia version entry. Dependencies may have been resolved with a different julia version.
└ @ ~/myRepository/julia/Manifest.toml:0
Precompiling project...
        Info Given MyProject was explicitly requested, output will be shown live 
[INFO @ someFile.c:103] Some info message.
  75 dependencies successfully precompiled in 109 seconds
  1 dependency had output during precompilation:
┌ MyProject
│  [Output was shown above]
└  
✔ [03m:20s] PackageCompiler: compiling nonincremental system image