PackageCompiler "create_app" function failing on slurm cluster: "error: undefined reference to 'ceil'"

Hi, I’m trying to figure out how best to troubleshoot this issue, but I’d like to use PackageCompiler to create a relocatable binary for collaborators. I upgraded PackageCompiler to v1.7.0 (on Julia v1.6.3) and that took care of one error (AssertionError: 0 < tid <= length(THREAD_RNGs) during compilation with PackageCompiler) but now I’m hitting another set (please see below). Due to the “error: undefined reference to ‘ceil’” errors I suspect it has something to do with access to gcc on my cluster, but I’ve certainly loaded gcc 8.2.0. It is also complicated by my desire to incorporate a python package using PyCall which throws a warning that it might not be relocatable, so I realize that could be related and I have some troubleshooting ahead of me.

Given these issues, I also tried compiling locally on my mac and a vagrant ubuntu VM. Both compiled the app just fine but both resulted in segmentation faults (first three lines of error pasted further below) when I tried to run them. Running the MyApp example from PackageCompiler was informative: it fails for the same reasons on the cluster (“undefined reference to ‘ceil’” and the like) but works just fine on my Vagrant VM

I’m probably dealing with at least two different issues, but I’d appreciate if anyone had any suggestions!

Error from slurm cluster:

julia> create_app(".julia/dev/myapp", "Compiledmyapp", force = true)
┌ Warning: Package PyCall has a build script, this might indicate that it is not relocatable
└ @ PackageCompiler ~/.julia/packages/PackageCompiler/r1Ahd/src/PackageCompiler.jl:734
┌ Warning: Package Conda has a build script, this might indicate that it is not relocatable
└ @ PackageCompiler ~/.julia/packages/PackageCompiler/r1Ahd/src/PackageCompiler.jl:734
[ Info: PackageCompiler: creating base system image (incremental=false)...
[ Info: PackageCompiler: creating system image object file, this might take a while...
[ Info: PackageCompiler: creating system image object file, this might take a while...
┌ Info: Compiling bit-parallel GC counter for LongSequence{<:NucleicAcidAlphabet}
└ @ BioSequences $HOME/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:9
┌ Info: Compiling bit-parallel mismatch counter for LongSequence{<:NucleicAcidAlphabet}
└ @ BioSequences $HOME/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:26
┌ Info: Compiling bit-parallel match counter for LongSequence{<:NucleicAcidAlphabet}
└ @ BioSequences $HOME/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:46
┌ Info: Compiling bit-parallel ambiguity counter...
└ @ BioSequences $HOME/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:73
┌ Info:         For a single LongSequence{<:NucleicAcidAlphabet}
└ @ BioSequences $HOME/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:74
┌ Info:         For a pair of LongSequence{<:NucleicAcidAlphabet}s
└ @ BioSequences $HOME/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:88
┌ Info: Compiling bit-parallel certainty counter for LongSequence{<:NucleicAcidAlphabet}
└ @ BioSequences $HOME/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:119
┌ Info: Compiling bit-parallel gap counter for LongSequence{<:NucleicAcidAlphabet}
└ @ BioSequences $HOME/.julia/packages/BioSequences/k4j4J/src/longsequences/counting.jl:140
../lib/julia/sys.so: error: undefined reference to 'ceil'
../lib/julia/sys.so: error: undefined reference to 'floor'
../lib/julia/sys.so: error: undefined reference to 'fma'
../lib/julia/sys.so: error: undefined reference to 'fmaf'
../lib/julia/sys.so: error: undefined reference to 'fmod'
../lib/julia/sys.so: error: undefined reference to 'pow'
../lib/julia/sys.so: error: undefined reference to 'rint'
../lib/julia/sys.so: error: undefined reference to 'rintf'
../lib/julia/sys.so: error: undefined reference to 'trunc'
collect2: error: ld returned 1 exit status
ERROR: failed process: Process(`gcc '-DJULIAC_PROGRAM_LIBNAME="../lib/julia/sys.so"' -m64 -march=x86-64 -o myapp $HOME/.julia/packages/PackageCompiler/r1Ahd/src/embedding_wrapper.c ../lib/julia/sys.so -O2 '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/../lib/julia' -std=gnu99 -I$HOME/julia-1.6.3/include/julia -fPIC -L$HOME/julia-1.6.3/lib -L$HOME/julia-1.6.3/lib/julia -Wl,--export-dynamic -Wl,-rpath-link,$HOME/julia-1.6.3/lib -Wl,-rpath-link,$HOME/julia-1.6.3/lib/julia -ljulia -ljulia-internal`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:525 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:440
  [3] run
    @ ./process.jl:438 [inlined]
  [4] run_compiler(cmd::Cmd)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/r1Ahd/src/PackageCompiler.jl:128
  [5] create_executable_from_sysimg(; sysimage_path::String, executable_path::String, c_driver_program_path::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/r1Ahd/src/PackageCompiler.jl:1102
  [6] (::PackageCompiler.var"#17#19"{String, String})()
    @ PackageCompiler ~/.julia/packages/PackageCompiler/r1Ahd/src/PackageCompiler.jl:1086
  [7] cd(f::PackageCompiler.var"#17#19"{String, String}, dir::String)
    @ Base.Filesystem ./file.jl:106
  [8] _create_app(package_dir::String, dest_dir::String, name::Nothing, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, audit::Bool, force::Bool, cpu_target::String; library_only::Bool, c_driver_program::String, julia_init_c_file::Nothing, header_files::Vector{String}, version::Nothing, compat_level::String, include_lazy_artifacts::Bool, sysimage_build_args::Cmd, include_transitive_dependencies::Bool)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/r1Ahd/src/PackageCompiler.jl:1083
  [9] #create_app#13
    @ ~/.julia/packages/PackageCompiler/r1Ahd/src/PackageCompiler.jl:819 [inlined]
 [10] top-level scope
    @ REPL[2]:1

Segmentation fault (first three lines) after compiling locally:

signal (11): Segmentation fault
in expression starting at none:0
memcpy at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)

Can you cd into Compiledmyapp and run

gcc '-DJULIAC_PROGRAM_LIBNAME="../lib/julia/sys.so"' -m64 -march=x86-64 -o myapp $HOME/.julia/packages/PackageCompiler/r1Ahd/src/embedding_wrapper.c ../lib/julia/sys.so -O2 '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/../lib/julia' -std=gnu99 -I$HOME/julia-1.6.3/include/julia -fPIC -L$HOME/julia-1.6.3/lib -L$HOME/julia-1.6.3/lib/julia -Wl,--export-dynamic -Wl,-rpath-link,$HOME/julia-1.6.3/lib -Wl,-rpath-link,$HOME/julia-1.6.3/lib/julia -ljulia -ljulia-internal -lm`

there? (I just added -lm to it).

Interesting, at first it gave me a gcc: error: ../lib/julia/sys.so: No such file or directory which was curious since I could find sys.io in $HOME/Compiledmyapp/lib/julia so I hardcoded the paths in that line you gave me and it ran and created an executable Compiledmyapp which I then moved to the bin directory. Running that did “work” but resulted in a segfault, but I think that is progress because this is the equivalent segfault to the one I get locally. So that’s good! :slight_smile:

Will keep hunting and thank you again for all your help!

So it looks like adding XAM.jl likely causes the segfaults: adding it to the MyApp example is enough to allow the compilation to complete but segfault once the compiled MyApp is run. I thought it was a PyCall problem due to the warnings but I can load PyCall and my package of choice without any segfault.

Realized I should be including the segmentation fault error from the MyApp example with XAM.jl added. This is after creating the executable using the code with the ‘-lm’ at the end.

Happy to create an issue for PackageCompiler.jl if you think that would be useful.

 is_crayons_loaded() = true                                                                                                                                                                                                                                                              
 Running the artifact                                                                                                                                                                                                                                                                    
 The result of 2*5^2 - 10 == 40.000000                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                         
 unsafe_string((Base.JLOptions()).image_file) = "$HOME/MyAppXAM/lib/julia/sys.so"                                                                                                                                                                                                        
 Example.domath(5) = 10                                                                                                                                                                                                                                                                  
 sin(0.0) = 0.0                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                         
 signal (11): Segmentation fault                                                                                                                                                                                                                                                         
 in expression starting at none:0                                                                                                                                                                                                                                                        
 __memmove_ssse3_back at /lib64/libc.so.6 (unknown line)                                                                                                                                                                                                                                 
 _ios_read at /buildworker/worker/package_linux64/build/src/support/ios.c:274 [inlined]                                                                                                                                                                                                  
 ios_readall at /buildworker/worker/package_linux64/build/src/support/ios.c:335                                                                                                                                                                                                          
 unsafe_read at ./iostream.jl:43                                                                                                                                                                                                                                                         
 unsafe_read at ./io.jl:722 [inlined]                                                                                                                                                                                                                                                    
 read! at ./io.jl:740 [inlined]                                                                                                                                                                                                                                                          
 rand! at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/RNGs.jl:53 [inlined]                                                                                                                                                                                                      
 rand! at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/Random.jl:267 [inlined]                                                                                                                                                                                                   
 rand at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/Random.jl:288 [inlined]                                                                                                                                                                                                    
 rand at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/Random.jl:291 [inlined]                                                                                                                                                                                                    
 make_seed at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/RNGs.jl:319                                                                                                                                                                                                           
 seed! at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/RNGs.jl:362 [inlined]                                                                                                                                                                                                     
 seed! at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/Random.jl:426 [inlined]                                                                                                                                                                                                   
 MersenneTwister at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/RNGs.jl:147                                                                                                                                                                                                     
 MersenneTwister at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/RNGs.jl:147 [inlined]                                                                                                                                                                                           
 default_rng at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/RNGs.jl:376                                                                                                                                                                                                         
 default_rng at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/RNGs.jl:370 [inlined]                                                                                                                                                                                               
 randstring at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Random/src/misc.jl:77                                                                                                                                                                                                           
 init_multi at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:1290                                                                                                                                                                                                 
 #addprocs#37 at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:438                                                                                                                                                                                                
 addprocs at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Distributed/src/cluster.jl:438 [inlined]                                                                                                                                                                                          
 #addprocs#245 at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Distributed/src/managers.jl:443 [inlined]                                                                                                                                                                                    
 addprocs at $HOME/julia-1.6.3/share/julia/stdlib/v1.6/Distributed/src/managers.jl:441 [inlined]                                                                                                                                                                                         
 real_main at $HOME/.julia/dev/MyApp/src/MyApp.jl:86                                                                                                                                                                                                                                     
 julia_main at $HOME/.julia/dev/MyApp/src/MyApp.jl:40                                                                                                                                                                                                                                    
 julia_main at ./none:127                                                                                                                                                                                                                                                                
 unknown function (ip: 0x2aaac62c254c)                                                                                                                                                                                                                                                   
 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]                                                                                                                                                                                                         
 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419                                                                                                                                                                                                             
 julia_main at $HOME/MyAppXAM/bin/../lib/julia/sys.so (unknown line)                                                                                                                                                                                                                     
 main at bin/MyApp (unknown line)                                                                                                                                                                                                                                                        
 __libc_start_main at /lib64/libc.so.6 (unknown line)                                                                                                                                                                                                                                    
 _start at bin/MyApp (unknown line)                                                                                                                                                                                                                                                      
 Allocations: 5985723 (Pool: 5984055; Big: 1668); GC: 6                                                                                                                                                                                                                                  
 Segmentation fault           

Can you try again now with the latest version (1.7.1), I think that issue should be fixed now.

I think it is! Upgrading to v1.7.1 no longer gives me seg fault errors, thank you! I still need to use your “-lm” code tweak in the compiled directory, but calling a simple bash script is getting me around that last step. Thank you again!

One hiccup that @kristoffer.carlsson might be interested in: I upgraded PackageCompiler to v1.7.2 and created my app, but I’m still hitting that error that requires me to use the “-lm” hack to compile it outside of Julia. Didn’t seem to be a big issue until I tried to move the folder to another location on my machine, at which point I got a segfault trying to run it (it runs fine in the original location).

Even removing the PyCall dependency (which had a relocatablility warning due to the build script) didn’t solve the issue. If it is a relocatability issue, should I be expecting that to result in a segfault?

It is still unclear to me why the -lm is needed, but I haven’t looked into it thoroughly yet. I don’t see why moving the app should make it segfault. Does it give any output?

It is weird, I take it the lm flag just links the math library that gcc uses, which is why I was hitting those weird errors for ‘ceil’ and the like. Do you think it is somehow no longer able to find the math library after I compile with -lm and move the dir? The only output I get when it segfaults trying to run myapp is:

signal (11): Segmentation fault
in expression starting at none:0
unknown function (ip: (nil))
main at bin/myapp (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at bin/myapp (unknown line)
Allocations: 3564 (Pool: 3549; Big: 15); GC: 0
Segmentation fault

Wait, I figured something out, I think segfaults go away and the binary is relocatable if I don’t screw up the gcc command: when adding “-lm” to the end of the failed process readout (failed process: Process(gcc '-DJULIAC_PROGRAM_LIBNAME…) I needed to edit a couple a path or two such as lib/julia/sys.so and I was too zealous about that. I think I might have hardcoded something there that shouldn’t have been there, but after fixing those paths the binary now seems to not segfault when I move it! (This is also with PackageCompiler v1.7.3, but I doubt that makes a difference in this case)

I guess the -lm mystery remains, and it would be nice to not have this manual step that people like me are prone to screwing up, but thank you again for your help!!

I agree. I opened an issue to track it: Should we link to libm? · Issue #601 · JuliaLang/PackageCompiler.jl · GitHub

Can you try with:

(run pkg> add PackageCompiler#kc/g++ to get that version of PackageCompiler)

Couldn’t find that branch but I’m assuming because you already merged it into master, so tried with master instead. Unfortunately no luck although the error message is different…

[ Info: PackageCompiler: creating base system image (incremental=false)...
[ Info: PackageCompiler: creating system image object file, this might take a while...
Precompiling project...
  36 dependencies successfully precompiled in 36 seconds
[ Info: PackageCompiler: creating system image object file, this might take a while...
/local/17551548/embedding_wrapper-7d83c6.o: In function `main':
embedding_wrapper.c:(.text+0x200): undefined reference to `julia_main'
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: failed process: Process(`clang '-DJULIAC_PROGRAM_LIBNAME="../lib/julia/sys.so"' -m64 -march=x86-64 -o myapp $HOME/.julia/packages/PackageCompiler/nlGB8/src/embedding_wrapper.c ../lib/julia/sys.so -O2 '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/../lib/julia' -std=gnu99 -I$HOME/julia-1.6.3/include/julia -fPIC -L$HOME/julia-1.6.3/lib -L$HOME/julia-1.6.3/lib/julia -Wl,--export-dynamic -Wl,-rpath-link$HOME/julia-1.6.3/lib -Wl,-rpath-link,$HOME/julia-1.6.3/lib/julia -ljulia -ljulia-internal`, ProcessExited(1)) [1]

(everything after this is just standard PackageCompiler “pipeline_error” stacktrace)

If you use the master branch of PackgeCompiler you need to look at Home · PackageCompiler. In your case, you need to add Base.@ccallable in front of your julia_main.

Or you can use PackageCompiler#release-1 which does not have any breaking changes.

1 Like

Sorry for the misunderstanding, switched to #release-1 as you suggested and it went back to the ../lib/julia/sys.so: error: undefined reference to 'ceil' errors as before. Next I’m going to try using the v2.0 master branch with the Base.@ccallable modification and see if that changes anything.

I realized I hadn’t push the change to that branch yet, heh. Will update

1 Like

Haha, that at least explains that, and I’m happy to report the master (after making the Base.@ccallable modification to my code) does not error and appears to run and relocate flawlessly, so that’s awesome! :slight_smile:

Thank you again!

1 Like

Great!