# Configuring MPI.jl to run on XC-40 / KNL with craype and cray-mpich

**URL:** <https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197>\
**Category:** Julia at Scale\
**Created:** [January 28, 2019, 10:20pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197 "2019-01-28T22:20:16Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![frankwillmore](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@frankwillmore](https://discourse.julialang.org/u/frankwillmore)\
**Post date:** [January 28, 2019, 10:20pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/1 "2019-01-28T22:20:16Z")

</div>

I am attempting to configure the MPI.jl package to run in Julia 1.0.3 on theta, a Cray XC-40/KNL. Cross-compilation has not been an issue, but interfacing to the CrayPE and Cray-MPICH are currently stumping me. I have been able to install Julia 1.0.3 and run non-MPI codes both interactively and non-interactively. I have also been able to add BinDep. In attempting to :

pkg\> add MPI

I have attempted to set up the environment according to the instructions at:

[https://github.com/JuliaParallel/MPI.jl#overriding-the-auto-detected-mpi-version](https://github.com/JuliaParallel/MPI.jl#overriding-the-auto-detected-mpi-version)

And invoke the repl with

CC=(which gcc) CXX=(which g++) FC=$(which gfortran) julia

as described there. I ran the craype wrappers using the -craype-verbose flag on an MPI ‘hello world’ in order to deconstruct how gcc and gfortran are to be invoked and to link MPI within that environment. I used this information to extract and provide values for JULIA\_MPI\_C\_LIBRARIES, JULIA\_MPI\_C\_INCLUDE\_PATH, JULIA\_MPI\_Fortran\_INCLUDE\_PATH, and JULIA\_MPI\_Fortran\_LIBRARIES as described in the documentation at:

[https://github.com/JuliaParallel/MPI.jl#overriding-the-auto-detected-mpi-version](https://github.com/JuliaParallel/MPI.jl#overriding-the-auto-detected-mpi-version)

The output from these was:

```julia
cc -shared -craype-verbose mpi_hello.c 

gcc -march=knl -shared -fPIC -D __CRAYXC -D__ CRAY_MIC_KNL -D __CRAYXT_COMPUTE_LINUX_TARGET -D__ TARGET_LINUX __mpi_hello.c -I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14__ g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1 __ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__ g34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26 __g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__ g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18 __g5196236.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__ g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14 __g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include -L/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/rca/2.2.18-6.0.6.0_19.14__ g2aa4f39.ari/lib64 -L/lib64 -lrca -lz -Wl,--as-needed,-lsci_gnu_71_mpi,--no-as-needed -Wl,--as-needed,-lsci_gnu_71,--no-as-needed -Wl,--as-needed,-lmpich_gnu_71,--no-as-needed -Wl,--as-needed,-lgfortran,-lquadmath,--no-as-needed -Wl,--as-needed,-lmvec,--no-as-needed -Wl,--as-needed,-lm,--no-as-needed -Wl,--as-needed,-lpthread,--no-as-needed

```

And:

```julia
ftn -shared -craype-verbose mpi_hello.f90 

gfortran -march=knl -shared -fPIC -D __CRAYXC -D__ CRAY_MIC_KNL -D __CRAYXT_COMPUTE_LINUX_TARGET -D__ TARGET_LINUX __mpi_hello.f90 -I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14__ g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1 __ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__ g34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26 __g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__ g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18 __g5196236.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__ g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14 __g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include -L/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/rca/2.2.18-6.0.6.0_19.14__ g2aa4f39.ari/lib64 -L/lib64 -lrca -lz -Wl,--as-needed,-lsci_gnu_71_mpi,--no-as-needed -Wl,--as-needed,-lsci_gnu_71,--no-as-needed -Wl,--as-needed,-lmpich_gnu_71,--no-as-needed -Wl,--as-needed,-lmpichf90_gnu_71,--no-as-needed -Wl,--as-needed,-lgfortran,-lquadmath,--no-as-needed -Wl,--as-needed,-lpthread,--no-as-needed

```

From these, I extracted and provided values to be picked up by CMake:

```julia
export JULIA_MPI_C_LIBRARIES="-I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include/ -L/opt/cray/pe/lib64 lmpich_gnu_71"

export JULIA_MPI_C_INCLUDE_PATH="-I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14 __g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1__ ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1 __g34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26__ g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12 __g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18__ g5196236.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8 __g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14__ g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include"

export JULIA_MPI_Fortran_INCLUDE_PATH="-I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14__

g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1 __ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__ g34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26

__g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__ g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18__g5196236.ari/include -I/o

pt/cray/wlm_detect/1.3.2-6.0.6.0_3.8 __g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14__ g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include"

export JULIA_MPI_Fortran_LIBRARIES="-L/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/lib64 -L/lib64 -lrca -lz -Wl,--as-needed,-lsci_gnu_71_mpi,--no-as-needed -Wl,--as-needed,-lsci_gnu_71,--no-as-needed -Wl,--as-needed,-lmpich_gnu_71,--no-as-needed -Wl,--as-needed,-lmpichf90_gnu_71,--no-as-needed -Wl,--as-needed,-lgfortran,-lquadmath,--no-as-needed -Wl,--as-needed,-lpthread,--no-as-needed"

```

When attempting to pkg\> add MPI, the build.log file in my depot reflects the following:

```julia
-- The Fortran compiler identification is GNU 7.3.0

-- The C compiler identification is GNU 7.3.0

-- Check for working Fortran compiler: /opt/gcc/7.3.0/bin/gfortran

-- Check for working Fortran compiler: /opt/gcc/7.3.0/bin/gfortran -- works

-- Detecting Fortran compiler ABI info

-- Detecting Fortran compiler ABI info - done

-- Checking whether /opt/gcc/7.3.0/bin/gfortran supports Fortran 90

-- Checking whether /opt/gcc/7.3.0/bin/gfortran supports Fortran 90 -- yes

-- Check for working C compiler: /opt/gcc/7.3.0/bin/gcc

-- Check for working C compiler: /opt/gcc/7.3.0/bin/gcc -- works

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Detecting C compile features

-- Detecting C compile features - done

-- Found Git: /usr/bin/git (found version "2.12.3") 

-- Found MPI_C: -I/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include/ -L/opt/cray/pe/lib64 -lmpich_gnu_71  

-- Found MPI_Fortran: -L/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/dmapp/default/lib64

 -L/opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/lib -L/opt/cray/rca/2.2.18-6.0.6.0_19.14__g2aa4f39.ari/lib64 -L/lib64 -lrca -lz -Wl,--as-needed,-lsci_gnu_71_mpi,--no-as-needed -Wl,--as-

needed,-lsci_gnu_71,--no-as-needed -Wl,--as-needed,-lmpich_gnu_71,--no-as-needed -Wl,--as-needed,-lmpichf90_gnu_71,--no-as-needed -Wl,--as-needed,-lgfortran,-lquadmath,--no-as-neede

d -Wl,--as-needed,-lpthread,--no-as-needed  

-- Detecting Fortran/C Interface

-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling

-- Looking for MPI_Comm_c2f

-- Looking for MPI_Comm_c2f - not found

-- Configuring done

-- Generating done

-- Build files have been written to: /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build

Scanning dependencies of target gen_constants

Scanning dependencies of target gen_constants

[11%] Building Fortran object CMakeFiles/gen_constants.dir/gen_constants.f90.o

f951: Warning: Nonexistent include directory '/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/-I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86_64/include -I/opt/cray/pe/mpt/7.7.3/g

ni/mpich-gnu/7.1/include -I/opt/cray/rca/2.2.18-6.0.6.0_19.14 __g2aa4f39.ari/include -I/opt/cray/alps/6.6.1-6.0.6.1_4.1__ ga6396bb.ari/include -I/opt/cray/xpmem/2.2.14-6.0.6.0_10.1__g

34333c9.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.6.0_3.26 __g527b6e1.ari/include -I/opt/cray/pe/pmi/5.0.14/include -I/opt/cray/ugni/6.0.14-6.0.6.0_18.12__ g777707d.ari/include -I/opt/cray/udreg/2.3.2-6.0.6.0_15.18 __g5196236.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__ g388ccd5.ari/include -I/opt/cray/krca/2.2.4-6.0.6.0_8.14__g8505b97.ari/include -I/opt/cray-hss-devel/8.0.0/include' [-Wmissing-include-dirs]

/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/gen_constants.f90:3: Error: Can't open included file 'mpif.h'

CMakeFiles/gen_constants.dir/build.make:62: recipe for target 'CMakeFiles/gen_constants.dir/gen_constants.f90.o' failed

make[2]: *** [CMakeFiles/gen_constants.dir/gen_constants.f90.o] Error 1

CMakeFiles/Makefile2:241: recipe for target 'CMakeFiles/gen_constants.dir/all' failed

make[1]: *** [CMakeFiles/gen_constants.dir/all] Error 2

Makefile:149: recipe for target 'all' failed

make: *** [all] Error 2

[ Info: Attempting to create directory /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build

[ Info: Changing directory to /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build

ERROR: LoadError: failed process: Process(`make`, ProcessExited(2)) [2]

Stacktrace:

 [1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42

 [2] pipeline_error at ./process.jl:705 [inlined]

 [3] #run#503(::Bool, ::Function, ::Cmd) at ./process.jl:663

 [4] run(::Cmd) at ./process.jl:661

 [5] run(::BinDeps.SynchronousStepCollection) at /home/willmore/.julia/packages/BinDeps/ZEval/src/BinDeps.jl:521 (repeats 3 times)

 [6] satisfy!(::BinDeps.LibraryDependency, ::Array{DataType,1}) at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:944

 [7] satisfy!(::BinDeps.LibraryDependency) at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:922

 [8] top-level scope at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:977

 [9] include at ./boot.jl:317 [inlined]

 [10] include_relative(::Module, ::String) at ./loading.jl:1044

 [11] include(::Module, ::String) at ./sysimg.jl:29

 [12] include(::String) at ./client.jl:392

 [13] top-level scope at none:0

in expression starting at /home/willmore/.julia/packages/MPI/U5ujD/deps/build.jl:54

```

What is troubling here in particular is the line:  
f951: Warning: Nonexistent include directory '/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/-I/opt/cray/pe/libsci/18.07.1/GNU/7.1/x86\_64/include -I/opt/cray/pe/mpt/7.7.3/g

Which seems to indicate that the first -I include directory entry is somehow being appended verbatim to the end of the deps directory, which seems inconsistent with how this information should be handled, at least as described in the MPI.jl directions.

So my questions are these:

- Should I have.a different understanding of how this information is to be interpreted?
- Has something changed in MPI.jl since this was documented?
- Is CMake misinterpreting the configuration information?
- Is there something else fundamentally missing?

We’re very interested in making MPI.jl available to run on theta, but are currently stuck at this point. I know this post is verbose. Thank you for reading.

---

<div class="post-metadata">

**Author:** ![favba](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/favba/32/2735_2.png) [@favba](https://discourse.julialang.org/u/favba)\
**Post date:** [January 28, 2019, 10:52pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/2 "2019-01-28T22:52:16Z")

</div>

> [@frankwillmore](#):
>
> export JULIA\_MPI\_Fortran\_INCLUDE\_PATH="-I/opt/cray/pe/libsci/

This is a wild guess, but you could try adding a space at the beginning of that export string:

```julia
export JULIA_MPI_Fortran_INCLUDE_PATH=" -I/opt/cray/pe/libsci/...

```

I noticed that there is a space missing here:

> [@frankwillmore](#):
>
> .julia/packages/MPI/U5ujD/deps/-I/opt/cray/pe/libsci/

---

<div class="post-metadata">

**Author:** ![frankwillmore](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@frankwillmore](https://discourse.julialang.org/u/frankwillmore)\
**Post date:** [January 29, 2019, 1:39am UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/3 "2019-01-29T01:39:14Z")

</div>

Hi favba–

Thanks for looking at it. There is no space before the ‘-I’ in the published MPI.jl instructions. When I do omit that space, the compiler still not find mpi.h:

```julia
 27 /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/gen_functions.c:4:10: fatal error: mpi.h: No such file or directory
 28 #include "mpi.h"
 29 ^ ~~~~~~

```

even though the correct mpi.h is actually available in the second -I location, i.e. /opt/cray/pe/mpt/7.7.3/gni/mpich-gnu/7.1/include

So perhaps to refine my question a bit: How do I _correctly_ pass this include location?

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [January 29, 2019, 5:07am UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/4 "2019-01-29T05:07:48Z")

</div>

@frankwillmore Please keep us updated on progress. I bet Julia goes like gangbusters on Knights Landing.

---

<div class="post-metadata">

**Author:** ![JaredCrean2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jaredcrean2/32/3574_2.png) [@JaredCrean2](https://discourse.julialang.org/u/JaredCrean2)\
**Post date:** [January 29, 2019, 2:00pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/5 "2019-01-29T14:00:25Z")

</div>

Looking at [this page](http://www.nersc.gov/users/computational-systems/cori/programming/compiling-codes-on-cori/), it looks like the correct compilers to on Cray are `cc` for C, `ftn` for Fortran, and `CC` for C++. Can you confirm if these exist on your system? If so, the first thing I would try is

```julia
CC=`which cc` FC=`which ftn` CXX=`which CC` julia

```

and then try to build MPI.jl (without setting any of the `JULIA_MPI_*` variables).

Something weird is happening with the `JULIA_MPI_*` variables, but the most direct path to getting MPI.jl built may be to avoid needing them.

Out of curiosity, what version of CMake are you using?

---

<div class="post-metadata">

**Author:** ![frankwillmore](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@frankwillmore](https://discourse.julialang.org/u/frankwillmore)\
**Post date:** [January 29, 2019, 8:29pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/6 "2019-01-29T20:29:03Z")

</div>

Hi John-

Appreciate your optimism. I will post updates.

---

<div class="post-metadata">

**Author:** ![frankwillmore](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@frankwillmore](https://discourse.julialang.org/u/frankwillmore)\
**Post date:** [January 29, 2019, 8:38pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/7 "2019-01-29T20:38:36Z")

</div>

Hi Jared–

Cmake version is 3.5.2. Cray wraps compilers as cc, CC and ftn. The actual command and args they execute can be had from specifying option -craype-verbose.

Running with only the command-line compiler specifications and setting no JULIA\_MPI\_\* vars causes CMake not to find MPI\_C info.

build.log:

```julia
-- The Fortran compiler identification is GNU 7.3.0
-- The C compiler identification is GNU 7.3.0
-- Check for working Fortran compiler: /opt/gcc/7.3.0/bin/gfortran
-- Check for working Fortran compiler: /opt/gcc/7.3.0/bin/gfortran -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /opt/gcc/7.3.0/bin/gfortran supports Fortran 90
-- Checking whether /opt/gcc/7.3.0/bin/gfortran supports Fortran 90 -- yes
-- Check for working C compiler: /opt/gcc/7.3.0/bin/gcc
-- Check for working C compiler: /opt/gcc/7.3.0/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Git: /usr/bin/git (found version "2.12.3") 
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find MPI_C (missing: MPI_C_LIBRARIES MPI_C_INCLUDE_PATH)
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindMPI.cmake:614 (find_package_handle_standard_args)
  CMakeLists.txt:5 (find_package)

-- Configuring incomplete, errors occurred!
See also "/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build/CMakeFiles/CMakeOutput.log".
[ Info: Attempting to create directory /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build
[ Info: Changing directory to /gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/build
ERROR: LoadError: failed process: Process(`cmake -DCMAKE_INSTALL_PREFIX=/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/src -DCMAKE_LIB_INSTALL_PREFIX=/gpfs/mira-home/willmore/.julia/packages/MPI/U5ujD/deps/usr/lib ..`, ProcessExited(1)) [1]
Stacktrace:
 [1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
 [2] pipeline_error at ./process.jl:705 [inlined]
 [3] #run#503(::Bool, ::Function, ::Cmd) at ./process.jl:663
 [4] run(::Cmd) at ./process.jl:661
 [5] run(::BinDeps.SynchronousStepCollection) at /home/willmore/.julia/packages/BinDeps/ZEval/src/BinDeps.jl:521 (repeats 3 times)
 [6] satisfy!(::BinDeps.LibraryDependency, ::Array{DataType,1}) at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:944
 [7] satisfy!(::BinDeps.LibraryDependency) at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:922
 [8] top-level scope at /home/willmore/.julia/packages/BinDeps/ZEval/src/dependencies.jl:977
 [9] include at ./boot.jl:317 [inlined]
 [10] include_relative(::Module, ::String) at ./loading.jl:1044
 [11] include(::Module, ::String) at ./sysimg.jl:29
 [12] include(::String) at ./client.jl:392
 [13] top-level scope at none:0
in expression starting at /home/willmore/.julia/packages/MPI/U5ujD/deps/build.jl:54

```

---

<div class="post-metadata">

**Author:** ![JaredCrean2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jaredcrean2/32/3574_2.png) [@JaredCrean2](https://discourse.julialang.org/u/JaredCrean2)\
**Post date:** [January 29, 2019, 9:41pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/8 "2019-01-29T21:41:34Z")

</div>

Interesting, thanks for posting the output. I think the next step is to try a version of CMake 3.10 or newer. There is some discussion on this Deal.II [thread](https://github.com/dealii/dealii/issues/5510) about how FindMPI changed in CMake 3.10 to use the compiler wrappers as the first source for finding MPI implementations.

The backstory is that MPI.jl adds the `MPI_*` variables to CMake on the command line by adding `-D` switches, so if the paths are getting set incorrectly the problem is likely internal to CMake. The code that does this is [here](https://github.com/JuliaParallel/MPI.jl/blob/master/deps/build.jl#L18) if you are curious.

---

<div class="post-metadata">

**Author:** ![frankwillmore](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@frankwillmore](https://discourse.julialang.org/u/frankwillmore)\
**Post date:** [January 29, 2019, 10:29pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/9 "2019-01-29T22:29:28Z")

</div>

Thanks for the tip on the change in CMake. Indeed, I tried with CMake 3.11.4 and now it’s not finding MPI\_C or MPI\_Fortran but missing a _different_ set of variables from last time. These variables also seem not to match those in build.jl:

```julia
 -- Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_HEADER_DIR MPI_C_WORKS) 
│ -- Could NOT find MPI_Fortran (missing: MPI_Fortran_LIB_NAMES MPI_Fortran_F77_HEADER_DIR MPI_Fortran_MODULE_DIR MPI_Fortran_WORKS) 
│ CMake Error at /lus/theta-fs0/software/buildtools/cmake/3.11.4/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
│ Could NOT find MPI (missing: MPI_C_FOUND MPI_Fortran_FOUND)
│ Call Stack (most recent call first):
│ /lus/theta-fs0/software/buildtools/cmake/3.11.4/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
│ /lus/theta-fs0/software/buildtools/cmake/3.11.4/share/cmake-3.11/Modules/FindMPI.cmake:1663 (find_package_handle_standard_args)
│ CMakeLists.txt:5 (find_package)

```

I’m also trying to get some clarity from Cray about what values I should be feeding these variables, once I figure out which variables I should use.

So please, keep it coming. This is helping me find direction as I dig deeper.

---

<div class="post-metadata">

**Author:** ![Mark\_Dalton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mark_dalton/32/8760_2.png) [@Mark\_Dalton](https://discourse.julialang.org/u/Mark_Dalton)\
**Post date:** [June 5, 2019, 6:18pm UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/10 "2019-06-05T18:18:06Z")

</div>

Just to update this. We have 22/26 tests passing.

For tests in Julia MPI not passing:  
So briefly the 4 remaining tests with errors:

1. test\_cman\_julia.jl - Fixed the issue it was not run with aprun do to the test  
issues related to the if statement setting up the ‘mgr’ (the aprun)

2. test\_cman\_tcp.jl - Socket complaints, but the different node output was interleaved  
and stdout/stderr.

3. test\_spawn.jl - MPI\_Comm\_spawn tests worked, except one that does have a  
extra parameter. MPI\_Comm\_spawn\_multiple has the first parameter as ‘count’.

4. test\_test.jl` - fails the MPI\_Finalize

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [June 6, 2019, 2:56am UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197/12 "2019-06-06T02:56:12Z")

</div>

Can you please open these as issues/pull requests on MPI.jl?