I am attempting to configure the MPI.jl package to run in Julia 1.0.3 on theta, a Cray XC-40/KNL. Cross-compilation has not been an issue, but interfacing to the CrayPE and Cray-MPICH are currently stumping me. I have been able to install Julia 1.0.3 and run non-MPI codes both interactively and non-interactively. I have also been able to add BinDep. In attempting to :
pkg> add MPI
I have attempted to set up the environment according to the instructions at:
https://github.com/JuliaParallel/MPI.jl#overriding-the-auto-detected-mpi-version
And invoke the repl with
CC=(which gcc) CXX=(which g++) FC=$(which gfortran) julia
as described there. I ran the craype wrappers using the -craype-verbose flag on an MPI ‘hello world’ in order to deconstruct how gcc and gfortran are to be invoked and to link MPI within that environment. I used this information to extract and provide values for JULIA_MPI_C_LIBRARIES, JULIA_MPI_C_INCLUDE_PATH, JULIA_MPI_Fortran_INCLUDE_PATH, and JULIA_MPI_Fortran_LIBRARIES as described in the documentation at:
https://github.com/JuliaParallel/MPI.jl#overriding-the-auto-detected-mpi-version
The output from these was:
cc -shared -craype-verbose mpi_hello.c
gcc -march=knl -shared -fPIC -D__CRAYXC -D__CRAY_MIC_KNL -D__CRAYXT_COMPUTE_LINUX_TARGET -D__TARGET_LINUX__ mpi_hello.c -I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1__ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__g34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26__g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18__g5196236.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14__g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include -L/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/lib64 -L/lib64 -lrca -lz -Wl,--as-needed,-lsci_gnu_71_mpi,--no-as-needed -Wl,--as-needed,-lsci_gnu_71,--no-as-needed -Wl,--as-needed,-lmpich_gnu_71,--no-as-needed -Wl,--as-needed,-lgfortran,-lquadmath,--no-as-needed -Wl,--as-needed,-lmvec,--no-as-needed -Wl,--as-needed,-lm,--no-as-needed -Wl,--as-needed,-lpthread,--no-as-needed
And:
ftn -shared -craype-verbose mpi_hello.f90
gfortran -march=knl -shared -fPIC -D__CRAYXC -D__CRAY_MIC_KNL -D__CRAYXT_COMPUTE_LINUX_TARGET -D__TARGET_LINUX__ mpi_hello.f90 -I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1__ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__g34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26__g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18__g5196236.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14__g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include -L/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/lib64 -L/lib64 -lrca -lz -Wl,--as-needed,-lsci_gnu_71_mpi,--no-as-needed -Wl,--as-needed,-lsci_gnu_71,--no-as-needed -Wl,--as-needed,-lmpich_gnu_71,--no-as-needed -Wl,--as-needed,-lmpichf90_gnu_71,--no-as-needed -Wl,--as-needed,-lgfortran,-lquadmath,--no-as-needed -Wl,--as-needed,-lpthread,--no-as-needed
From these, I extracted and provided values to be picked up by CMake:
export JULIA_MPI_C_LIBRARIES="-I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include/ -L/opt/cray/pe/lib64 lmpich_gnu_71"
export JULIA_MPI_C_INCLUDE_PATH="-I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1__ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__g34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26__g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18__g5196236.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14__g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include"
export JULIA_MPI_Fortran_INCLUDE_PATH="-I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14__
g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1__ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__g34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26
__g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18__g5196236.ari/include -I/o
pt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14__g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include"
export JULIA_MPI_Fortran_LIBRARIES="-L/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/lib64 -L/lib64 -lrca -lz -Wl,--as-needed,-lsci_gnu_71_mpi,--no-as-needed -Wl,--as-needed,-lsci_gnu_71,--no-as-needed -Wl,--as-needed,-lmpich_gnu_71,--no-as-needed -Wl,--as-needed,-lmpichf90_gnu_71,--no-as-needed -Wl,--as-needed,-lgfortran,-lquadmath,--no-as-needed -Wl,--as-needed,-lpthread,--no-as-needed"
When attempting to pkg> add MPI, the build.log file in my depot reflects the following:
-- The Fortran compiler identification is GNU 7.3.0
-- The C compiler identification is GNU 7.3.0
-- Check for working Fortran compiler: /opt/gcc/7.3.0/bin/gfortran
-- Check for working Fortran compiler: /opt/gcc/7.3.0/bin/gfortran -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /opt/gcc/7.3.0/bin/gfortran supports Fortran 90
-- Checking whether /opt/gcc/7.3.0/bin/gfortran supports Fortran 90 -- yes
-- Check for working C compiler: /opt/gcc/7.3.0/bin/gcc
-- Check for working C compiler: /opt/gcc/7.3.0/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Git: /usr/bin/git (found version "2.12.3")
-- Found MPI_C: -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include/ -L/opt/cray/pe/lib64 -lmpich_gnu_71
-- Found MPI_Fortran: -L/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/dmapp/default/lib64
-L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/lib64 -L/lib64 -lrca -lz -Wl,--as-needed,-lsci_gnu_71_mpi,--no-as-needed -Wl,--as-
needed,-lsci_gnu_71,--no-as-needed -Wl,--as-needed,-lmpich_gnu_71,--no-as-needed -Wl,--as-needed,-lmpichf90_gnu_71,--no-as-needed -Wl,--as-needed,-lgfortran,-lquadmath,--no-as-neede
d -Wl,--as-needed,-lpthread,--no-as-needed
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Looking for MPI_Comm_c2f
-- Looking for MPI_Comm_c2f - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build
Scanning dependencies of target gen_constants
Scanning dependencies of target gen_constants
[ 11%] Building Fortran object CMakeFiles/gen_constants.dir/gen_constants.f90.o
f951: Warning: Nonexistent include directory '/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/-I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/g
ni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1__ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__g
34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26__g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18__g5196236.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14__g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include' [-Wmissing-include-dirs]
/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/gen_constants.f90:3: Error: Can't open included file 'mpif.h'
CMakeFiles/gen_constants.dir/build.make:62: recipe for target 'CMakeFiles/gen_constants.dir/gen_constants.f90.o' failed
make[2]: *** [CMakeFiles/gen_constants.dir/gen_constants.f90.o] Error 1
CMakeFiles/Makefile2:241: recipe for target 'CMakeFiles/gen_constants.dir/all' failed
make[1]: *** [CMakeFiles/gen_constants.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2
[ Info: Attempting to create directory /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build
[ Info: Changing directory to /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build
ERROR: LoadError: failed process: Process(`make`, ProcessExited(2)) [2]
Stacktrace:
[1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
[2] pipeline_error at ./process.jl:705 [inlined]
[3] #run#503(::Bool, ::Function, ::Cmd) at ./process.jl:663
[4] run(::Cmd) at ./process.jl:661
[5] run(::BinDeps.SynchronousStepCollection) at /home/willmore/.julia/packages/BinDeps/ZEval/src/BinDeps.jl:521 (repeats 3 times)
[6] satisfy!(::BinDeps.LibraryDependency, ::Array{DataType,1}) at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:944
[7] satisfy!(::BinDeps.LibraryDependency) at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:922
[8] top-level scope at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:977
[9] include at ./boot.jl:317 [inlined]
[10] include_relative(::Module, ::String) at ./loading.jl:1044
[11] include(::Module, ::String) at ./sysimg.jl:29
[12] include(::String) at ./client.jl:392
[13] top-level scope at none:0
in expression starting at /home/willmore/.julia/packages/MPI/U5ujD/deps/build.jl:54
What is troubling here in particular is the line:
f951: Warning: Nonexistent include directory '/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/-I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/g
Which seems to indicate that the first -I include directory entry is somehow being appended verbatim to the end of the deps directory, which seems inconsistent with how this information should be handled, at least as described in the MPI.jl directions.
So my questions are these:
- Should I have.a different understanding of how this information is to be interpreted?
- Has something changed in MPI.jl since this was documented?
- Is CMake misinterpreting the configuration information?
- Is there something else fundamentally missing?
We’re very interested in making MPI.jl available to run on theta, but are currently stuck at this point. I know this post is verbose. Thank you for reading.