# ERROR: could not load library libcutensor.so
**URL:** https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020
**Category:** GPU
**Created:** [July 14, 2020, 7:19am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020 "2020-07-14T07:19:15Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 14, 2020, 7:19am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/1 "2020-07-14T07:19:15Z")
After a fresh installation of CUDA.jl 1.1/1.0, we got the following error on a cluster when trying to precompile a package that has CUDA in its dependencies:
```julia
[Info: Precompiling ParallelStencil [94395366-693c-11ea-3b26-d9b7aac5d958]
Downloading artifact: CUDNN_CUDA102
Downloading artifact: CUTENSOR_CUDA102
ERROR: LoadError: LoadError: LoadError: could not load library "/home/lraess/.julia/artifacts/fbe34931d3c1bebd56fbc2edba0f8ece5295fed7/lib/libcutensor.so"
/lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/lraess/.julia/artifacts/fbe34931d3c1bebd56fbc2edba0f8ece5295fed7/lib/libcutensor.so)
Stacktrace:
```
NOTE 1: the package does not use cutensors.
NOTE 2: CUDA was installed as follows:
```julia
module load CUDA/10.0
export JULIA_CUDA_USE_BINARYBUILDER=false
julia
] add CUDA
```
We would appreciate quick help very much as we need to run something there for the JuliaCon video!
Thanks!!
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [July 14, 2020, 9:50am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/2 "2020-07-14T09:50:04Z")
What version of glibc do you have on the cluster? Looks something pretty old?
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 14, 2020, 10:39am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/3 "2020-07-14T10:39:01Z")
Thanks for your reply @giordano. Yes, it looks like the `libc.so.6` that is found is too old. There is another one available on the cluster and when I set the path to it in LD\_LIBRARY\_PATH there is no more `libc.so.6`-error given when doing ldd on `libcutensor.so`.
However with that exported julia cannot be started: it segfaults at startup. Also defining the LD\_LIBRARY\_PATH inside julia using `ENV` did not work. It gave the following error:
```julia
julia> ENV["LD_LIBRARY_PATH"] = "/soft/glibc/glibc-2.17/lib:$(ENV["LD_LIBRARY_PATH"])"
"/soft/glibc/glibc-2.17/lib:/soft/glibc/glibc-2.17/lib:\$LD_LIBRARY_PATH"
julia> using ParallelStencil
[Info: Precompiling ParallelStencil [94395366-693c-11ea-3b26-d9b7aac5d958]
ERROR: IOError: write: broken pipe (EPIPE)
Stacktrace:
[1] uv_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:953
[2] unsafe_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:1007
[3] write(::Base.PipeEndpoint, ::String) at ./strings/io.jl:183
[4] create_expr_cache(::String, ::String, ::Array{Pair{Base.PkgId,UInt64},1}, ::Base.UUID) at ./loading.jl:1176
[5] compilecache(::Base.PkgId, ::String) at ./loading.jl:1261
[6] _require(::Base.PkgId) at ./loading.jl:1029
[7] require(::Base.PkgId) at ./loading.jl:927
[8] require(::Module, ::Symbol) at ./loading.jl:922
```
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [July 14, 2020, 10:58am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/4 "2020-07-14T10:58:24Z")
> [@samo](#):
>
> Also defining the LD\_LIBRARY\_PATH inside julia using `ENV` did not work. It gave the following error:
Yeah, I wouldn’t expect changing this variable inside Julia to be too much helpful. Out of curiosity, do you have any CUDA module available on the cluster? What version? Maybe you can try with an older version of CUDA provided by Julia.
CC: @maleadt
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [July 14, 2020, 11:19am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/5 "2020-07-14T11:19:05Z")
> [@giordano](#):
>
> Out of curiosity, do you have any CUDA module available on the cluster? What version?
If that’s the case you can use `JULIA_CUDA_USE_BINARYBUILDER=false`.
If you want to keep using artifacts, you can use an older version that does not provide CUTENSOR by specifying `JULIA_CUDA_VERSION=...`. Or you could disable the CUTENSOR version check in ` __runtime_init__ ` (in CUDA.jl’s `initialization.jl`), it’s that check that triggers use of the library.
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [July 14, 2020, 11:24am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/6 "2020-07-14T11:24:15Z")
Actually, the snippet at the top is already setting `JULIA_CUDA_USE_BINARYBUILDER=false` 🤔
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [July 14, 2020, 11:25am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/7 "2020-07-14T11:25:09Z")
Probably only during `Pkg.add`, and the value isn’t cached anywhere currently.
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 14, 2020, 11:29am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/8 "2020-07-14T11:29:14Z")
> [@maleadt](#):
>
> Probably only during `Pkg.add` , and the value isn’t cached anywhere currently.
We exported it before Pkg.add, yes. Does one need to export it before every run then (at least when precompilation happens)?
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [July 14, 2020, 11:29am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/9 "2020-07-14T11:29:50Z")
The environment variable needs to be defined like that every time CUDA.jl is loaded, as it looks for artifacts whenever the module is initialized.
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 14, 2020, 11:30am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/10 "2020-07-14T11:30:20Z")
Oh, I see, this clarifies a lot. I will try…
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 14, 2020, 11:40am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/11 "2020-07-14T11:40:49Z")
Setting `JULIA_CUDA_USE_BINARYBUILDER=false` also for running (including precompiling) solved the problem as it does not anymore use artifacts.
BTW: I think it should be hightlighted in the doc [here](https://juliagpu.gitlab.io/CUDA.jl/installation/overview/#Local-installation-1) that `JULIA_CUDA_USE_BINARYBUILDER=false` must not only be set for `Pkg.add`/`Pkg.build`.
Thanks!!
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 14, 2020, 11:48am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/12 "2020-07-14T11:48:46Z")
@maleadt, a small related question: do I assume right that CUDA-aware MPI will not be able to work when one uses artifacts, right? For CUDA-aware MPI, one needs to have a CUDA-aware MPI installation and build CUDA.jl and MPI.jl with the CUDA and MPI used for the CUDA-aware MPI installation, right?
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [July 14, 2020, 11:59am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/13 "2020-07-14T11:59:04Z")
I don’t have experience with CUDA aware MPI + different CUDA toolkits (maybe @vchuravy or @simonbyrne do) . Generally the toolkit is backwards comptible though.
---
### Author:  [@vchuravy](https://discourse.julialang.org/u/vchuravy)
#### Post date: [July 14, 2020, 4:04pm UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/14 "2020-07-14T16:04:00Z")
Yes for CUDA-aware MPI you should use CUDA and MPI provided by your cluster administrator.
```julia
export JULIA_MPI_BINARY=system
export JULIA_CUDA_USE_BINARYBUILDER=false
```
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 14, 2020, 4:30pm UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/15 "2020-07-14T16:30:21Z")
Thanks @vchuravy. This is what we have done. I am just trying to understand if it in general it was also possible to use CUDA-aware MPI with artifacts as I know multiple people getting started with small clusters / multi-GPU desktops with Julia, GPU and MPI… So to get started, or when a quick temporary solution is needed it could be nice to be able to use CUDA-aware MPI with artifacts…
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [July 14, 2020, 5:52pm UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/16 "2020-07-14T17:52:47Z")
Seconding this. AFAIK CUDA-aware MPI is the only way to do multi-gpu sync on a desktop machine/local workstation, but unlike in a cluster it’s almost certainly not installed and I haven’t been able to find binaries anywhere…
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 20, 2020, 2:17pm UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/17 "2020-07-20T14:17:47Z")
> [@samo](#):
>
> The environment variable needs to be defined like that every time CUDA.jl is loaded, as it looks for artifacts whenever the module is initialized.
@maleadt, it would seem more intuitive to me that one needs to specify `JULIA_CUDA_USE_BINARYBUILDER=false` only at installation time. BTW: I think that is what @simonbyrne meant in [this issue](https://github.com/JuliaGPU/CUDA.jl/issues/204) when he wrote " It would also be useful to have a mechanism to save these preferences so that they persist between sessions/versions ([FFTW.jl](https://github.com/JuliaMath/FFTW.jl/blob/master/deps/build.jl) and [MPI.jl](https://github.com/JuliaParallel/MPI.jl/blob/master/deps/build.jl) save their preferences to a file in `.julia/prefs/`".
I also believe that this would be an improvement - or is there anything that speak against it?
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [July 20, 2020, 3:51pm UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/18 "2020-07-20T15:51:54Z")
There is some discussion of this issue here: [https://github.com/JuliaGPU/CUDA.jl/issues/204](https://github.com/JuliaGPU/CUDA.jl/issues/204)
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [July 23, 2020, 1:20pm UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/19 "2020-07-23T13:20:48Z")
> [@samo](#):
>
> I also believe that this would be an improvement - or is there anything that speak against it?
I was waiting for Pkg support before doing so: [Add `Preferences` subsystem by staticfloat · Pull Request #1835 · JuliaLang/Pkg.jl · GitHub](https://github.com/JuliaLang/Pkg.jl/pull/1835)
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [July 24, 2020, 6:24am UTC](https://discourse.julialang.org/t/error-could-not-load-library-libcutensor-so/43020/20 "2020-07-24T06:24:26Z")
> [@maleadt](#):
>
> I was waiting for Pkg support before doing so: [Add `Preferences` subsystem by staticfloat · Pull Request #1835 · JuliaLang/Pkg.jl · GitHub](https://github.com/JuliaLang/Pkg.jl/pull/1835)
Sure, that makes sense.