# Juliacon21-gpu\_workshop notebook \`could not load library "libcuda.so.1"\`

**URL:** <https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014>\
**Category:** GPU\
**Created:** [July 21, 2021, 4:34am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014 "2021-07-21T04:34:06Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![banksiaboy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/banksiaboy/32/27032_2.png) [@banksiaboy](https://discourse.julialang.org/u/banksiaboy)\
**Post date:** [July 21, 2021, 4:34am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/1 "2021-07-21T04:34:06Z")

</div>

I am trying to get started on the JuliaCon GPU tutorial. I have tried everything I can think of

I’m on `Ubuntu 20.04.2 LTS`.

If I use CUDA from the repl, it is successful.  
If I try it from a jupyter notebook, launched from the same repl that successfully uses CUDA, I get  
`could not load library "libcuda.so.1"`

The only instance of that filename is:

```julia
% locate libcuda.so.1          
/usr/lib/x86_64-linux-gnu/libcuda.so.1

```

Which is not in the nvidia install of CUDA. It is from

```julia
% dpkg -S /usr/lib/x86_64-linux-gnu/libcuda.so.1                                              
libnvidia-compute-470:amd64: /usr/lib/x86_64-linux-gnu/libcuda.so.1

```

which is maintained by ubuntu core developers. I have reported a bug there. But I don’t really know why this doesn’t work. There goes another day…

The version of CUDA being used should be managed here →

```julia
sudo update-alternatives --display cuda
cuda - auto mode
  link best version is /usr/local/cuda-11.4
  link currently points to /usr/local/cuda-11.4
  link cuda is /usr/local/cuda
/usr/local/cuda-11.4 - priority 114

```

My system is otherwise well-behaved AFAIK

```julia
% nvidia-smi
...
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 
...
% nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jun__2_19:15:15_PDT_2021
Cuda compilation tools, release 11.4, V11.4.48
Build cuda_11.4.r11.4/compiler.30033411_0

```

The Julia repl is running CUDA fine:

```julia
julia> using CUDA

julia> CUDA.versioninfo()
CUDA toolkit 11.3.1, artifact installation
CUDA driver 11.4.0
NVIDIA driver 470.57.2

Libraries: 
- CUBLAS: 11.5.1
- CURAND: 10.2.4
- CUFFT: 10.4.2
- CUSOLVER: 11.1.2
- CUSPARSE: 11.6.0
- CUPTI: 14.0.0
- NVML: 11.0.0+470.57.2
- CUDNN: 8.20.0 (for CUDA 11.3.0)
- CUTENSOR: 1.3.0 (for CUDA 11.2.0)

Toolchain:
- Julia: 1.6.1
- LLVM: 11.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

1 device:
  0: NVIDIA GeForce GTX 1050 Ti (sm_61, 3.446 GiB / 3.938 GiB available)

```

```julia
% julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.6.1 (2021-04-23)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

(@v1.6) pkg> add CUDA, IJulia
    Updating registry at `~/.julia/registries/General`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.6/Project.toml`
  No Changes to `~/.julia/environments/v1.6/Manifest.toml`

shell> ls
common_definitions.jl FunWithArrays.ipynb Introduction_CUDA.ipynb Lilly_hat.jpg sneak_peek
DeviceSideRNG.ipynb ImageProcessing.ipynb JuliaSet.ipynb Manifest.toml src
Diffusion.ipynb Introduction_AMDGPU.ipynb kernelabstractions Project.toml

julia> using IJulia

julia> jupyterlab()

```

I get this error:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/4/4/44a7a09163e55de96a3643859ea7266c895a478b.png)

---

<div class="post-metadata">

**Author:** ![banksiaboy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/banksiaboy/32/27032_2.png) [@banksiaboy](https://discourse.julialang.org/u/banksiaboy)\
**Post date:** [July 21, 2021, 4:36am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/2 "2021-07-21T04:36:52Z")

</div>

I am guessing it is both of the Jupyter developers, and Ubuntu developers struggling with “update-alternatives”. The Julia repl seems to get the right result.

---

<div class="post-metadata">

**Author:** ![banksiaboy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/banksiaboy/32/27032_2.png) [@banksiaboy](https://discourse.julialang.org/u/banksiaboy)\
**Post date:** [July 21, 2021, 6:14am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/3 "2021-07-21T06:14:35Z")

</div>

The scenario works in Pluto, but not in Jupyter

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/f/ff9ded4b2bce4f5e83bfd23aac045d84bf8e88b8.png)

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 21, 2021, 7:28am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/4 "2021-07-21T07:28:51Z")

</div>

`libcuda.so` is expected to be provided by the NVIDIA driver, aka `libnvidia-*`. `update-alternatives cuda` has nothing to do with this. And if the library is correctly detected from a command line, there was probably no need to file a bug with Ubuntu developers.

The only question is why `libcuda.so` in `/usr/lib/x86_64-linux-gnu` isn’t discovered in Jupyter. That path should have been registered by an entry in `/etc/ld.so.conf.d`. Are you sure you’re running your notebook on the same server? It’s also possible that your system is badly configured, e.g. without an entry for `/usr/lib/x86_64-linux-gnu` in `/etc/ld.so.conf.d`, and that you used a `LD_LIBRARY_PATH` entry to discover `libcuda.so` – that environment variable may have gotten lost by the Jupyter environment.

---

<div class="post-metadata">

**Author:** ![banksiaboy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/banksiaboy/32/27032_2.png) [@banksiaboy](https://discourse.julialang.org/u/banksiaboy)\
**Post date:** [July 21, 2021, 7:48am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/5 "2021-07-21T07:48:51Z")

</div>

Just got it working in an LXC container with CUDA pass-through.  
All the CUDA libraries seem to be behaving themselves with some mixed-parentage…

```julia
% locate 'libcuda.so'
/usr/lib/x86_64-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcuda.so.1
/usr/lib/x86_64-linux-gnu/libcuda.so.470.57.02
/usr/local/cuda-11.4/targets/x86_64-linux/lib/stubs/libcuda.sosudo update

% sudo update-alternatives --list cuda
/usr/local/cuda-11.4

% locate nvcc        
/usr/local/cuda-11.4/bin/nvcc
/usr/local/cuda-11.4/bin/nvcc.profile

NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4

```

I should probably update paths or alternatives, but I’m over my head here, and it works…

something like the following ubuntu 20.04.2 commands:

```julia
sudo apt install pip aptitude
pip install jill --user -U

```

added Ubuntu [PPA for graphic-drivers](https://linuxhint.com/ubuntu_nvidia_ppa/) and installed the recommended driver.

added the [repository for NVIDIA drivers](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html), but did not install.

```julia
% sudo apt update
% sudo apt install build-essential
% sudo apt-get install linux-headers-$(uname -r) # nvidia prep advice from nvidia

```

Then I searched for cuda-toolkit, and chose the version that matched the CUDA version installed in the host - visible within the container with `nvidia-smi`

```julia
% sudo aptitude search cuda-toolkit
[...] - CUDA Toolkit 11.2 meta-package                                        
p cuda-toolkit-11-3 - CUDA Toolkit 11.3 meta-package                                        
p cuda-toolkit-11-3-config-common - Common config package for CUDA Toolkit 11.3.                          
i cuda-toolkit-11-4 - CUDA Toolkit 11.4 meta-package                                        
[...]   

```

The 1.4 meta-package matches.

```julia
% sudo aptitude install cuda-toolkit

```

packages were installed from all the added repositories, CUDA was available in both Pluto and Jupyter.

I managed to find the conda package installed by Julia, and activated to let me run

```julia
% jupyter notebook list               
Currently running servers:
http://localhost:8888/?token=<blah-blah> :: /home/ubuntu

```

I had sshd into the container with port forwarding to allow access to jupyter

```julia
ssh -L localhost:8888:localhost:8888 $c1.lxd

```

Note:  
CUDA is installed in the base Ubuntu operating system, and is passed through to the container - no install required.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 21, 2021, 7:52am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/6 "2021-07-21T07:52:27Z")

</div>

You don’t need the CUDA toolkit, CUDA.jl won’t use it and downloads its own copy. You don’t need the graphics-drivers PPA. Just add the NVIDIA one, and install the `cuda-drivers` metapackage, that’s all you need to do.

> [@banksiaboy](#):
>
> CUDA is installed in the base Ubuntu operating system, and is passed through to the container - no install required.

No, the NVIDIA driver is passed through (confusingly providing `libcuda.so`), CUDA itself isn’t. But you don’t need CUDA.

---

<div class="post-metadata">

**Author:** ![banksiaboy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/banksiaboy/32/27032_2.png) [@banksiaboy](https://discourse.julialang.org/u/banksiaboy)\
**Post date:** [July 22, 2021, 7:59am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/7 "2021-07-22T07:59:01Z")

</div>

Thanks very much for your reply.  
When you say ‘the NVIDIA one’, it isn’t clear to me.  
What you mean - either driver or it’s provenance?  
Ubuntu 20.04 has nvidia drivers, alternatives are also available from nvidia.

I had no trouble I was aware of directly in the repl. It only worked in the Jupiter notebook installed by the repl with the the packages from Ubuntu-drivers. Success at that stage may have been dumb luck.

There are so many moving parts.

Is there somewhere I can find a specification of the specific prerequisite packages?

Cheers…

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 22, 2021, 8:02am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/8 "2021-07-22T08:02:53Z")

</div>

> [@banksiaboy](#):
>
> When you say ‘the NVIDIA one’, it isn’t clear to me.  
> What you mean - either driver or it’s provenance?

It’s always the NVIDIA driver you need, there is no alternative (e.g. `nouveau` doesn’t work). So I meant the NVIDIA apt repository, because it typically provides more up-to-date drivers.

> [@banksiaboy](#):
>
> There are so many moving parts.
> 
> Is there somewhere I can find a specification of the specific prerequisite packages?

There aren’t that many, the only thing you need is the NVIDIA driver module and the corresponding CUDA driver library (`libcuda.so`). Typically, those are installed together using any of the driver installers from the NVIDIA home page. But since you’re using Ubuntu, I recommend using the NVIDIA apt repository and just installing `cuda-drivers`. You can use the official Ubuntu one as well, it works equally well, but might be a little older.

---

<div class="post-metadata">

**Author:** ![banksiaboy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/banksiaboy/32/27032_2.png) [@banksiaboy](https://discourse.julialang.org/u/banksiaboy)\
**Post date:** [July 22, 2021, 12:15pm UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/9 "2021-07-22T12:15:29Z")

</div>

I very much appreciate how Julia tries to encapsulate and configure its own dependencies.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 22, 2021, 12:23pm UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/10 "2021-07-22T12:23:01Z")

</div>

If only we could distribute `libcuda.so` too 🙂 It’s actually possible, but only on datacenter hardware, so I haven’t bothered to implement that feature yet. And you’d still need to install a driver, it would just simplify upgrading.

---

<div class="post-metadata">

**Author:** ![banksiaboy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/banksiaboy/32/27032_2.png) [@banksiaboy](https://discourse.julialang.org/u/banksiaboy)\
**Post date:** [July 23, 2021, 12:14am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/11 "2021-07-23T00:14:14Z")

</div>

Hi again,

I have spent two days re-configuring system to try and get this working.

It think this issue is related:[libcuda.so.1 is missing with conda env tf-gpu #11743](https://github.com/ContinuumIO/anaconda-issues/issues/11743)

I believe my evidence demonstrates the jupyter notebook using julia package `IJulia` is not using the same method to find `libcuda ` as the repl.

in the repl:

```julia-auto
j% julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.6.2 (2021-07-14)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using Libdl

julia> Libdl.find_library("libcuda")
"libcuda"

```

In the notebook with a freshly reset kernel:

![image](https://global.discourse-cdn.com/julialang/original/3X/6/e/6e0434bfbc5ba93e4aeb01560c8a48fa1d6c53b5.png)

## With jupyterlab installed locally

```julia-auto
% conda create -n jupyter-julia python=3.9
% conda activate jupyter-julia
% mamba install -c conda-forge jupyterlab 

```

Check julia in this environment

```julia-auto
(jupyter-julia) 
% julia --version
julia version 1.6.2
julia> using Libdl
julia> Libdl.find_library("libcuda")
julia> using Libdl

julia> Libdl.find_library("libcuda")
"libcuda"

julia> using CUDA

julia> CUDA.versioninfo()
CUDA toolkit 11.3.1, artifact installation
CUDA driver 11.4.0
NVIDIA driver 470.57.2

Libraries: 
- CUBLAS: 11.5.1
- CURAND: 10.2.4
- CUFFT: 10.4.2
- CUSOLVER: 11.1.2
- CUSPARSE: 11.6.0
- CUPTI: 14.0.0
- NVML: 11.0.0+470.57.2
- CUDNN: 8.20.0 (for CUDA 11.3.0)
- CUTENSOR: 1.3.0 (for CUDA 11.2.0)

Toolchain:
- Julia: 1.6.2
- LLVM: 11.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

1 device:
  0: NVIDIA GeForce GTX 1050 Ti (sm_61, 3.337 GiB / 3.938 GiB available)

```

From the newly installed Jupyter system with a 1.6 kernel,

```julia-auto
using Libdl
Libdl.find_library("libcuda")
""

```

---

<div class="post-metadata">

**Author:** ![banksiaboy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/banksiaboy/32/27032_2.png) [@banksiaboy](https://discourse.julialang.org/u/banksiaboy)\
**Post date:** [July 23, 2021, 12:39am UTC](https://discourse.julialang.org/t/juliacon21-gpu-workshop-notebook-could-not-load-library-libcuda-so-1/65014/12 "2021-07-23T00:39:06Z")

</div>

I have reported a bug against IJulia  
[#1015](https://github.com/JuliaLang/IJulia.jl/issues/1015)