Singularity, ERROR: Unable to find compatible target in system image

Specific Domains Julia at Scale
1

I’m trying to compile a Singularity container for running on a HPC cluster.

My container works on my laptop (where I build it), but when I transfer it over to the cluster, I get a:

ERROR: Unable to find compatible target in system image.

Does someone know what the build instructions are to make this work. I’ve tried

I’ve tried make --cpu_target=generic, but that does not seem to be a recognized option?

thanks
(from the singularity def build file)

        git clone https://github.com/JuliaLang/julia.git
        cd /julia
        git checkout v1.5.4
        ## from ERROR: Unable to find compatible target in system image.
        make -j 8 --cpu-target=generic 
        mkdir /opt/julia 
        export JULIA_DEPOT_PATH=/opt/julia
        /julia/julia -e 'using Pkg ; Pkg.add("SpecialFunctions")'
        /julia/julia -e 'using Pkg ; Pkg.add("BSON")'
        /julia/julia -e 'using Pkg ; Pkg.add("LinearAlgebra")'
        /julia/julia -e 'using Pkg ; Pkg.add("Printf")'
        /julia/julia -e 'using SpecialFunctions; using BSON ; using LinearAlgebra ; using Printf'
2

Solved… this seems to fix the Singularity build, Julia runs inside Singularity on the HPC cluster.

export JULIA_CPU_TARGET='generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)'

from https://docs.julialang.org/en/v1/devdocs/sysimg/

Next question, Is there a better way to build the packages I need? (I.e some magic with PackageCompiler?).

        ## from Singularity def build file
        ## build Julia from github source
        git clone https://github.com/JuliaLang/julia.git
        cd /julia
        git checkout v1.5.4
        ## from ERROR: Unable to find compatible target in system image. 
        ## this seams to solve it
        export JULIA_CPU_TARGET='generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)'
        make -j 8
        mkdir /opt/julia
        export JULIA_DEPOT_PATH=/opt/julia
        /julia/julia -e 'using Pkg ; Pkg.add("SpecialFunctions")'
        /julia/julia -e 'using Pkg ; Pkg.add("BSON")'
        /julia/julia -e 'using Pkg ; Pkg.add("LinearAlgebra")'
        /julia/julia -e 'using Pkg ; Pkg.add("Printf")'
        /julia/julia -e 'using SpecialFunctions; using BSON ; using LinearAlgebra ; using Printf'
1 Like
3

You may be interested in GitHub - JuliaContainerization/SimpleContainerGenerator.jl: Create container images for using Julia packages (especially useful in environments without Internet access)

It generates Docker images, but it is very easy to convert Docker images to Singularity images. I routinely take the Docker images generated by SimpleContainerGenerator.jl, convert them to Singularity images, and run them on HPC clusters.

4

That looks awesome! Thanks! (strange that didn’t come up in my google searches.) I’m glad I struggled to burn my own from scratch (good learning experience), but once is plenty…

I have access to an academic HPC cluster, but not for long. Do you have any experience running Julia/Singularity containers in the google Cloud, AWS, or other commercial cloud vendors? Any tips would be greatly appreciated. Now that I can actually get my code running, I’m curious the best way to get results back from the cloud (database, git repos, etc).

thanks again