# CUDA: CUDA driver not found

**URL:** <https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186>\
**Category:** GPU\
**Tags:** question\
**Created:** [July 18, 2024, 1:56pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186 "2024-07-18T13:56:57Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 18, 2024, 1:56pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/1 "2024-07-18T13:56:57Z")

</div>

A few days ago the IT-department of my company added a virtual GPU to my virtual machine which I would like to use in Julia.

I use julia 1.10.3 (which I can load as a module through (`module load julia/1.10.3`) and have added CUDA v.5.4.2 to my environment.

I could locate the CUDA driver library though

```julia
$ ldconfig -p | grep libcuda.so
        libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
        libcuda.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so 

```

and also

```julia
$ nvidia-smi |grep NVIDIA
        | NVIDIA-SMI 535.161.08 Driver Version: 535.161.08 CUDA Version: 12.2 |

```

looks ok to me.  
However the CUDA driver cannot be found

```julia
$ JULIA_DEBUG=CUDA_Driver_jll julia --project -e "using CUDA; CUDA.versioninfo()"
┌ Debug: No system CUDA driver found
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:104
ERROR: CUDA driver not found
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] functional
   @ ~/.julia/packages/CUDA/75aiI/src/initialization.jl:24 [inlined]
 [3] versioninfo(io::Base.TTY)
   @ CUDA ~/.julia/packages/CUDA/75aiI/src/utilities.jl:42
 [4] top-level scope
   @ none:1

```

When investigating further I found that could not load the shared CUDA driver library:

```julia-repl
julia> using Base.Libc.Libdl

julia> Libdl.dlopen("/lib/x86_64-linux-gnu/libcuda.so")
ERROR: could not load library "/lib/x86_64-linux-gnu/libcuda.so"
/lib/x86_64-linux-gnu/libcuda.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:117
 [2] dlopen
   @ ./libdl.jl:116 [inlined]
 [3] dlopen(s::String)
   @ Base.Libc.Libdl ./libdl.jl:116
 [4] top-level scope
   @ REPL[1]:1

```

So wondered what the dependencies of the CUDA driver library are

```julia
$ ldd "/lib/x86_64-linux-gnu/libcuda.so"
        linux-vdso.so.1 (0x00007fff7f1de000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd2e0d92000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd2e0b6a000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd2e0b65000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd2e0b60000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd2e0b5b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd2e2afe000)

```

and checked whether all of them are already loaded. They are all loaded!

```julia-repl
julia> required_libraries = [ 
       "linux-vdso.so.1"
       "/lib/x86_64-linux-gnu/libm.so.6"
       "/lib/x86_64-linux-gnu/libc.so.6"
       "/lib/x86_64-linux-gnu/libdl.so.2"
       "/lib/x86_64-linux-gnu/libpthread.so.0"
       "/lib/x86_64-linux-gnu/librt.so.1"
       "/lib64/ld-linux-x86-64.so.2"
       ];

julia> loaded_libraries = Libdl.dllist();

julia> for lib in required_libraries
           @assert lib in loaded_libraries
       end

julia>

```

So as a last resort I tried to `strace -o log.txt ./test.jl` the following file:

```julia
#!/usr/bin/env -S julia --color=yes --startup-file=no --project
using Base.Libc.Libdl
Libdl.dlopen("/lib/x86_64-linux-gnu/libcuda.so")

```

However I can’t really make sense of the output; Can somebody help me with that?

```julia
execve("./test.jl", ["./test.jl"], 0x7fff94997190 /* 64 vars */) = 0
brk(NULL) = 0x5628211e8000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffc84173580) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14f4063000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=86471, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 86471, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f14f404d000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\237\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0 \0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 48, 848) = 48
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\244;\374\204(\337f#\315I\214\234\f\256\271\32"..., 68, 896) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2216304, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2260560, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f14f3e25000
mmap(0x7f14f3e4d000, 1658880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f14f3e4d000
mmap(0x7f14f3fe2000, 360448, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x7f14f3fe2000
mmap(0x7f14f403a000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x214000) = 0x7f14f403a000
mmap(0x7f14f4040000, 52816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f14f4040000
close(3) = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14f3e22000
arch_prctl(ARCH_SET_FS, 0x7f14f3e22740) = 0
set_tid_address(0x7f14f3e22a10) = 602416
set_robust_list(0x7f14f3e22a20, 24) = 0
rseq(0x7f14f3e230e0, 0x20, 0, 0x53053053) = 0
mprotect(0x7f14f403a000, 16384, PROT_READ) = 0
mprotect(0x562820d1e000, 4096, PROT_READ) = 0
mprotect(0x7f14f409d000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7f14f404d000, 86471) = 0
getrandom("\xd0\xf5\x41\x73\x7d\x58\x1a\xaa", 8, GRND_NONBLOCK) = 8
brk(NULL) = 0x5628211e8000
brk(0x562821209000) = 0x562821209000
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=3048928, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 3048928, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f14f3b39000
close(3) = 0
execve("/mycompany/apps/bin/julia_1.10.3/julia", ["julia", "--color=yes", "--startup-file=no", "--project", "./test.jl"], 0x7ffc84173768 /* 64 vars */) = -1 ENOEXEC (Exec format error)
execve("/bin/sh", ["/bin/sh", "/mycompany/apps/bin/julia_1.10.3/julia", "--color=yes", "--startup-file=no", "--project", "./test.jl"], 0x7ffc84173768 /* 64 vars */) = 0
brk(NULL) = 0x56328345d000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd46a0e670) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb9a9860000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=86471, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 86471, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb9a984a000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\237\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0 \0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 48, 848) = 48
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\244;\374\204(\337f#\315I\214\234\f\256\271\32"..., 68, 896) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2216304, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2260560, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb9a9622000
mmap(0x7fb9a964a000, 1658880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7fb9a964a000
mmap(0x7fb9a97df000, 360448, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x7fb9a97df000
mmap(0x7fb9a9837000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x214000) = 0x7fb9a9837000
mmap(0x7fb9a983d000, 52816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb9a983d000
close(3) = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb9a961f000
arch_prctl(ARCH_SET_FS, 0x7fb9a961f740) = 0
set_tid_address(0x7fb9a961fa10) = 602416
set_robust_list(0x7fb9a961fa20, 24) = 0
rseq(0x7fb9a96200e0, 0x20, 0, 0x53053053) = 0
mprotect(0x7fb9a9837000, 16384, PROT_READ) = 0
mprotect(0x5632824d2000, 8192, PROT_READ) = 0
mprotect(0x7fb9a989a000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7fb9a984a000, 86471) = 0
getuid() = 166922
getgid() = 100513
getpid() = 602416
rt_sigaction(SIGCHLD, {sa_handler=0x5632824c8aa0, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER, sa_restorer=0x7fb9a9664520}, NULL, 8) = 0
geteuid() = 166922
getrandom("\x4b\x70\x24\xfc\xa3\xe7\x32\x9e", 8, GRND_NONBLOCK) = 8
brk(NULL) = 0x56328345d000
brk(0x56328347e000) = 0x56328347e000
getppid() = 602413
newfstatat(AT_FDCWD, "/path/to/the/working/directory", {st_mode=S_IFDIR|0755, st_size=9, ...}, 0) = 0
newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0755, st_size=9, ...}, 0) = 0
openat(AT_FDCWD, "/mycompany/apps/bin/julia_1.10.3/julia", O_RDONLY) = 3
fcntl(3, F_DUPFD, 10) = 10
close(3) = 0
fcntl(10, F_SETFD, FD_CLOEXEC) = 0
geteuid() = 166922
getegid() = 100513
rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x5632824c8aa0, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER, sa_restorer=0x7fb9a9664520}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER, sa_restorer=0x7fb9a9664520}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=SIG_DFL, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER, sa_restorer=0x7fb9a9664520}, NULL, 8) = 0
read(10, "singularity exec /mycompany/apps/contai"..., 8192) = 69
newfstatat(AT_FDCWD, "/mycompany/apps/bin/julia_1.10.3/singularity", 0x7ffd46a0e360, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/opt/anaconda3/bin/singularity", 0x7ffd46a0e360, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/mycompany/home/xyz/micromamba/condabin/singularity", 0x7ffd46a0e360, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/sbin/singularity", 0x7ffd46a0e360, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/bin/singularity", 0x7ffd46a0e360, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/sbin/singularity", 0x7ffd46a0e360, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/bin/singularity", {st_mode=S_IFREG|0755, st_size=42468904, ...}, 0) = 0
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], NULL, 8) = 0
vfork() = 602417
rt_sigprocmask(SIG_SETMASK, [], ~[KILL STOP RTMIN RT_1], 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 602417
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=602417, si_uid=166922, si_status=1, si_utime=4, si_stime=6} ---
rt_sigreturn({mask=[]}) = 602417
wait4(-1, 0x7ffd46a0e2bc, WNOHANG, NULL) = -1 ECHILD (No child processes)
read(10, "", 8192) = 0
exit_group(1) = ?
+++ exited with 1 +++

```

Any help or ideas where to look would be much appreciated!

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 18, 2024, 2:30pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/2 "2024-07-18T14:30:02Z")

</div>

Can you do `Libdl.dlopen("libcuda.so")` under `LD_DEBUG=libs`?

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 18, 2024, 5:14pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/3 "2024-07-18T17:14:00Z")

</div>

First, thank you for replying, I appreciate it!  
Starting the julia-repl with `LD_DEBUG=libs julia --project` yields:

```julia-repl
julia> using Base.Libc.Libdl

julia> Libdl.dlopen("libcuda.so")
    676486: find library=libcuda.so [0]; searching
    676486: search path=/.singularity.d/libs (LD_LIBRARY_PATH)
    676486: trying file=/.singularity.d/libs/libcuda.so
    676486: search path=/opt/julia-1.10.3/bin/../lib/julia:/opt/julia-1.10.3/bin/../lib/julia/.. (RUNPATH from file /opt/julia-1.10.3/bin/julia)
    676486: trying file=/opt/julia-1.10.3/bin/../lib/julia/libcuda.so
    676486: trying file=/opt/julia-1.10.3/bin/../lib/julia/../libcuda.so
    676486: search cache=/etc/ld.so.cache
    676486: search path=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/lib:/usr/lib (system search path)
    676486: trying file=/lib/x86_64-linux-gnu/libcuda.so
    676486: trying file=/usr/lib/x86_64-linux-gnu/libcuda.so
    676486: trying file=/lib/libcuda.so
    676486: trying file=/usr/lib/libcuda.so
    676486:
ERROR: could not load library "libcuda.so"
libcuda.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:117
 [2] dlopen
   @ ./libdl.jl:116 [inlined]
 [3] dlopen(s::String)
   @ Base.Libc.Libdl ./libdl.jl:116
 [4] top-level scope
   @ REPL[2]:1

julia>

```

Isn’t this strange, since the path is actually listed?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 18, 2024, 7:04pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/4 "2024-07-18T19:04:04Z")

</div>

OK, that wasn’t very interesting; maybe try `LD_DEBUG=all`? This will generate a lot of output.

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 18, 2024, 7:38pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/5 "2024-07-18T19:38:21Z")

</div>

this yields:

```julia-repl
julia> using Base.Libc.Libdl
    722495: symbol=localtime_r; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: symbol=localtime_r; lookup in file=/opt/julia-1.10.3/bin/../lib/libjulia.so.1.10 [0]
    722495: symbol=localtime_r; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libunwind.so.8 [0]
    722495: symbol=localtime_r; lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
    722495: symbol=localtime_r; lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
    722495: symbol=localtime_r; lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
    722495: symbol=localtime_r; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libz.so.1 [0]
    722495: symbol=localtime_r; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libatomic.so.1 [0]
    722495: symbol=localtime_r; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libstdc++.so.6 [0]
    722495: symbol=localtime_r; lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
    722495: symbol=localtime_r; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libgcc_s.so.1 [0]
    722495: symbol=localtime_r; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `localtime_r'
    722495: symbol=wcsftime; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: symbol=wcsftime; lookup in file=/opt/julia-1.10.3/bin/../lib/libjulia.so.1.10 [0]
    722495: symbol=wcsftime; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libunwind.so.8 [0]
    722495: symbol=wcsftime; lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
    722495: symbol=wcsftime; lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
    722495: symbol=wcsftime; lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
    722495: symbol=wcsftime; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libz.so.1 [0]
    722495: symbol=wcsftime; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libatomic.so.1 [0]
    722495: symbol=wcsftime; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libstdc++.so.6 [0]
    722495: symbol=wcsftime; lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
    722495: symbol=wcsftime; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libgcc_s.so.1 [0]
    722495: symbol=wcsftime; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `wcsftime'
    722495: symbol=ios_get_writable; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ios_get_writable'
    722495: symbol=ios_write; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ios_write'
    722495: symbol=ios_flush; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ios_flush'
    722495: symbol=ijl_array_del_beg; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_array_del_beg'
    722495: symbol=ijl_array_sizehint; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_array_sizehint'

julia> Libdl.dlopen("libcuda.so")
    722495:
    722495: file=libcuda.so [0]; dynamically loaded by /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: find library=libcuda.so [0]; searching
    722495: search path=/.singularity.d/libs (LD_LIBRARY_PATH)
    722495: trying file=/.singularity.d/libs/libcuda.so
    722495: search path=/opt/julia-1.10.3/bin/../lib/julia:/opt/julia-1.10.3/bin/../lib/julia/.. (RUNPATH from file /opt/julia-1.10.3/bin/julia)
    722495: trying file=/opt/julia-1.10.3/bin/../lib/julia/libcuda.so
    722495: trying file=/opt/julia-1.10.3/bin/../lib/julia/../libcuda.so
    722495: search cache=/etc/ld.so.cache
    722495: search path=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/lib:/usr/lib (system search path)
    722495: trying file=/lib/x86_64-linux-gnu/libcuda.so
    722495: trying file=/usr/lib/x86_64-linux-gnu/libcuda.so
    722495: trying file=/lib/libcuda.so
    722495: trying file=/usr/lib/libcuda.so
    722495:
    722495: symbol=ijl_get_excstack; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_get_excstack'
    722495: symbol=jl_get_fieldtypes; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `jl_get_fieldtypes'
    722495: symbol=ijl_type_intersection; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_type_intersection'
    722495: symbol=ijl_gc_pool_alloc; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_gc_pool_alloc'
    722495: symbol=ijl_lookup_code_address; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_lookup_code_address'
ERROR: 722495: symbol=ijl_specializations_lookup; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_specializations_lookup'
    722495: symbol=ijl_invoke_api; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_invoke_api'
    722495: symbol=ijl_ir_flag_inferred; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_ir_flag_inferred'
    722495: symbol=ijl_copy_code_info; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_copy_code_info'
    722495: symbol=ijl_field_index; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_field_index'
    722495: symbol=ijl_stored_inline; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_stored_inline'
    722495: symbol=ijl_instantiate_type_in_env; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_instantiate_type_in_env'
    722495: symbol=ijl_module_globalref; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_module_globalref'
    722495: symbol=ijl_isa_compileable_sig; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_isa_compileable_sig'
    722495: symbol=jl_method_get_table; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `jl_method_get_table'
    722495: symbol=ijl_normalize_to_compilable_sig; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_normalize_to_compilable_sig'
    722495: symbol=ijl_array_grow_at; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_array_grow_at'
    722495: symbol=ijl_string_ptr; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_string_ptr'
could not load library "libcuda.so"
libcuda.so: cannot open shared object file: No such file or directory
Stacktrace:
    722495: symbol=ijl_module_parent; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_module_parent'
 [1] 722495: symbol=ijl_uncompress_argnames; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_uncompress_argnames'
    722495: symbol=ijl_types_equal; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_types_equal'
    722495: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/libjulia.so.1.10 [0]
    722495: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libunwind.so.8 [0]
    722495: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
    722495: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
    722495: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
    722495: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libz.so.1 [0]
    722495: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libatomic.so.1 [0]
    722495: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libstdc++.so.6 [0]
    722495: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
    722495: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libgcc_s.so.1 [0]
    722495: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `strlen'
    722495: symbol=ijl_binding_resolved_p; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_binding_resolved_p'
    722495: symbol=jl_binding_owner; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `jl_binding_owner'
    722495: symbol=ijl_is_binding_deprecated; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_is_binding_deprecated'
dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ 722495: symbol=ijl_module_name; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    722495: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_module_name'
Base.Libc.Libdl ./libdl.jl:117
 [2] dlopen
   @ ./libdl.jl:116 [inlined]
 [3] dlopen(s::String)
   @ Base.Libc.Libdl ./libdl.jl:116
 [4] top-level scope
   @ REPL[2]:1

julia> 

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 18, 2024, 7:44pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/6 "2024-07-18T19:44:54Z")

</div>

> [@engram](#):
>
> `trying file=/lib/x86_64-linux-gnu/libcuda.so`

I would have expected some details after that line, on why the loader refused that library. Can you check if it’s a valid symlink to a valid library (e.g. doing `file /lib/x86_64-linux-gnu/libcuda.so.1`, assuming `libcuda.so` there points to `libcuda.so.1`)?

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 19, 2024, 7:02am UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/7 "2024-07-19T07:02:52Z")

</div>

I see. So `libcuda.so` is a symbolic link to `libcuda.so.1` which is a symbolic link pointing to `libcuda.so.535.161.08`:

```julia
$ file /lib/x86_64-linux-gnu/libcuda.so
       /lib/x86_64-linux-gnu/libcuda.so: symbolic link to libcuda.so.1
$ file /lib/x86_64-linux-gnu/libcuda.so.1
       /lib/x86_64-linux-gnu/libcuda.so.1: symbolic link to libcuda.so.535.161.08
$ file /lib/x86_64-linux-gnu/libcuda.so.535.161.08 
       /lib/x86_64-linux-gnu/libcuda.so.535.161.08: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=e80660c91b15804c3be20f39c07ae2744f70022d, stripped

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 19, 2024, 7:29am UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/8 "2024-07-19T07:29:37Z")

</div>

> [@engram](#):
>
> `/lib/x86_64-linux-gnu/libcuda.so.535.161.08`

Does `dlopen`ing that library directly work? If you have `root` permissions, try running `ldconfig` too.

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 19, 2024, 7:55am UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/9 "2024-07-19T07:55:22Z")

</div>

`dlopen`ing that library under `LD_DEBUG=all` leads to

```julia-repl
julia> Libdl.dlopen("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")
    970327:
    970327: file=/lib/x86_64-linux-gnu/libcuda.so.535.161.08 [0]; dynamically loaded by /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
ERROR: 970327: symbol=ijl_string_ptr; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    970327: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_string_ptr'
could not load library "/lib/x86_64-linux-gnu/libcuda.so.535.161.08"
/lib/x86_64-linux-gnu/libcuda.so.535.161.08: cannot open shared object file: No such file or directory
Stacktrace:
    970327: symbol=ijl_module_parent; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    970327: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_module_parent'
 [1] 970327: symbol=ijl_uncompress_argnames; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    970327: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_uncompress_argnames'
    970327: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    970327: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/libjulia.so.1.10 [0]
    970327: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libunwind.so.8 [0]
    970327: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
    970327: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
    970327: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
    970327: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libz.so.1 [0]
    970327: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libatomic.so.1 [0]
    970327: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libstdc++.so.6 [0]
    970327: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
    970327: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libgcc_s.so.1 [0]
    970327: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
    970327: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `strlen'
    970327: symbol=ijl_binding_resolved_p; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    970327: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_binding_resolved_p'
    970327: symbol=jl_binding_owner; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    970327: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `jl_binding_owner'
    970327: symbol=ijl_is_binding_deprecated; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    970327: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_is_binding_deprecated'
dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ 970327: symbol=ijl_module_name; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
    970327: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_module_name'
Base.Libc.Libdl ./libdl.jl:117
 [2] dlopen
   @ ./libdl.jl:116 [inlined]
 [3] dlopen(s::String)
   @ Base.Libc.Libdl ./libdl.jl:116
 [4] top-level scope
   @ REPL[3]:1

julia>

```

Also double checked, it has the dependencies mentioned above. Unfortunately I don’t have root permissions, but I’ll ask my IT department and let you know!

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 19, 2024, 10:26am UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/10 "2024-07-19T10:26:58Z")

</div>

ok, my it department has run `ldconfig`. Runing `ldconfig -p | grep libcuda.so` yields:

```julia
$ ldconfig -p |grep libcuda.so
        libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
        libcuda.so.1 (libc6) => /usr/lib32/libcuda.so.1
        libcuda.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so
        libcuda.so (libc6) => /usr/lib32/libcuda.so

```

Running again under `LD_DEBUG=all` results in:

```julia-repl
julia> using Base.Libc.Libdl
   1043662: symbol=ijl_array_sizehint; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_array_sizehint'

julia> Libdl.dlopen("libcuda.so")
   1043662:
   1043662: file=libcuda.so [0]; dynamically loaded by /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: find library=libcuda.so [0]; searching
   1043662: search path=/.singularity.d/libs (LD_LIBRARY_PATH)
   1043662: trying file=/.singularity.d/libs/libcuda.so
   1043662: search path=/opt/julia-1.10.3/bin/../lib/julia:/opt/julia-1.10.3/bin/../lib/julia/.. (RUNPATH from file /opt/julia-1.10.3/bin/julia)
   1043662: trying file=/opt/julia-1.10.3/bin/../lib/julia/libcuda.so
   1043662: trying file=/opt/julia-1.10.3/bin/../lib/julia/../libcuda.so
   1043662: search cache=/etc/ld.so.cache
   1043662: search path=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/lib:/usr/lib (system search path)
   1043662: trying file=/lib/x86_64-linux-gnu/libcuda.so
   1043662: trying file=/usr/lib/x86_64-linux-gnu/libcuda.so
   1043662: trying file=/lib/libcuda.so
   1043662: trying file=/usr/lib/libcuda.so
   1043662:
   1043662: symbol=ijl_get_excstack; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_get_excstack'
   1043662: symbol=ijl_lookup_code_address; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_lookup_code_address'
ERROR: 1043662: symbol=ijl_copy_code_info; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_copy_code_info'
   1043662: symbol=ijl_field_index; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_field_index'
   1043662: symbol=ijl_stored_inline; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_stored_inline'
   1043662: symbol=ijl_module_globalref; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_module_globalref'
   1043662: symbol=ijl_string_ptr; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_string_ptr'
could not load library "libcuda.so"
libcuda.so: cannot open shared object file: No such file or directory
Stacktrace:
   1043662: symbol=ijl_module_parent; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_module_parent'
 [1] 1043662: symbol=ijl_uncompress_argnames; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_uncompress_argnames'
   1043662: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/libjulia.so.1.10 [0]
   1043662: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libunwind.so.8 [0]
   1043662: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
   1043662: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
   1043662: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
   1043662: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libz.so.1 [0]
   1043662: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libatomic.so.1 [0]
   1043662: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libstdc++.so.6 [0]
   1043662: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
   1043662: symbol=strlen; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libgcc_s.so.1 [0]
   1043662: symbol=strlen; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `strlen'
   1043662: symbol=ijl_binding_resolved_p; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_binding_resolved_p'
   1043662: symbol=jl_binding_owner; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `jl_binding_owner'
   1043662: symbol=ijl_is_binding_deprecated; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_is_binding_deprecated'
dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ 1043662: symbol=ijl_module_name; lookup in file=/opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
   1043662: binding file /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0] to /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]: normal symbol `ijl_module_name'
Base.Libc.Libdl ./libdl.jl:117
 [2] dlopen
   @ ./libdl.jl:116 [inlined]
 [3] dlopen(s::String)
   @ Base.Libc.Libdl ./libdl.jl:116
 [4] top-level scope
   @ REPL[3]:1

julia> Libdl.dlopen("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")
   1043662:
   1043662: file=/lib/x86_64-linux-gnu/libcuda.so.535.161.08 [0]; dynamically loaded by /opt/julia-1.10.3/bin/../lib/julia/libjulia-internal.so.1.10 [0]
ERROR: could not load library "/lib/x86_64-linux-gnu/libcuda.so.535.161.08"
/lib/x86_64-linux-gnu/libcuda.so.535.161.08: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:117
 [2] dlopen
   @ ./libdl.jl:116 [inlined]
 [3] dlopen(s::String)
   @ Base.Libc.Libdl ./libdl.jl:116
 [4] top-level scope
   @ REPL[4]:1

julia> 

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 19, 2024, 11:15am UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/11 "2024-07-19T11:15:39Z")

</div>

That all looks fine; I have no idea why it fails to load. Can you try doing the same from C?

```c
#include <stdio.h>
#include <dlfcn.h>

int main() {
    const char *lib_path = "/lib/x86_64-linux-gnu/libcuda.so.535.161.08";
    void *handle;

    // Attempt to load the library
    handle = dlopen(lib_path, RTLD_LAZY);
    if (!handle) {
        fprintf(stderr, "Error: %s\n", dlerror());
        return 1;
    }

    printf("Library loaded successfully!\n");

    // Close the library
    if (dlclose(handle) != 0) {
        fprintf(stderr, "Error closing library: %s\n", dlerror());
        return 1;
    }

    printf("Library closed successfully!\n");
    return 0;
}

```

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 19, 2024, 12:20pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/12 "2024-07-19T12:20:51Z")

</div>

Thank you for the effort of writing that snippet!  
I saved the snippet as `load_library.c` and ran:

```julia
$ gcc -o load_library load_library.c
$ ./load_library 
     Library loaded successfully!
     Library closed successfully!
$

```

The gcc version is `gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)`

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 19, 2024, 3:14pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/13 "2024-07-19T15:14:36Z")

</div>

Can you confirm that the equivalent (`using Libdl; Libdl.dlopen("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")`) _doesn’t_ work from Julia, using a fresh session? If that’s the case, it probably warrants opening an issue on the Julia repository.

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 21, 2024, 11:02am UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/14 "2024-07-21T11:02:56Z")

</div>

Unfortunately yes, will create an issue! Thank you again for your time and effort!

```julia
$ module load julia/1.10.3
$ ls -ls /lib/x86_64-linux-gnu/libcuda.so.535.161.08
28696 -rwxr-xr-x 1 root root 29380752 Mar 6 01:04 /lib/x86_64-linux-gnu/libcuda.so.535.161.08
$ julia -e 'using Base.Libc.Libdl; dlopen("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")'
ERROR: could not load library "/lib/x86_64-linux-gnu/libcuda.so.535.161.08"
/lib/x86_64-linux-gnu/libcuda.so.535.161.08: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:117
 [2] dlopen
   @ ./libdl.jl:116 [inlined]
 [3] dlopen(s::String)
   @ Base.Libc.Libdl ./libdl.jl:116
 [4] top-level scope
   @ none:1
$

```

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 21, 2024, 1:03pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/15 "2024-07-21T13:03:02Z")

</div>

@engram can you try the same C program, but changing the flags to `dlopen` to `RTLD_LAZY|RTLD_DEEPBIND|RTLD_LOCAL`? According to [`dlopen`](https://docs.julialang.org/en/v1/stdlib/Libdl/#Base.Libc.Libdl.dlopen) docstring those should be the default flags on Linux. Just checking the flags aren’t screwing up the loading of the library. Conversely, try

```julia
using Base.Libc.Libdl; dlopen("/lib/x86_64-linux-gnu/libcuda.so.535.161.08", RTLD_LAZY)

```

in Julia.

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 22, 2024, 11:56am UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/16 "2024-07-22T11:56:36Z")

</div>

Changed the line in the C program to the following:

```julia
handle = dlopen(lib_path, RTLD_LAZY|RTLD_DEEPBIND|RTLD_LOCAL);

```

Still get the same output that the library can be loaded and closed in C.  
`dlopen`ing julia still gives me an error despite of the flag:

```julia-repl
julia> using Base.Libc.Libdl;

julia> dlopen("/lib/x86_64-linux-gnu/libcuda.so.535.161.08", RTLD_LAZY)
ERROR: could not load library "/lib/x86_64-linux-gnu/libcuda.so.535.161.08"
/lib/x86_64-linux-gnu/libcuda.so.535.161.08: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
   @ Base.Libc.Libdl ./libdl.jl:117
 [2] dlopen(s::String, flags::UInt32)
   @ Base.Libc.Libdl ./libdl.jl:116
 [3] top-level scope
   @ REPL[14]:1

julia>

```

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 22, 2024, 12:12pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/17 "2024-07-22T12:12:40Z")

</div>

Just realized that julia doesn’t realize that `/lib/x86_64-linux-gnu/libcuda.so.535.161.08` is a valid filepath:

```julia-repl
julia> ispath("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")
false

julia> ispath("/lib/x86_64-linux-gnu/libc.so.6")
true

```

although `libcuda.so.535.161.08` definitely exists:

```julia
$ ls -lsa /lib/x86_64-linux-gnu/libcuda.so.535.161.08 
28696 -rwxr-xr-x 1 root root 29380752 Mar 6 01:04 /lib/x86_64-linux-gnu/libcuda.so.535.161.08

$ ls -lsa /lib/x86_64-linux-gnu/libc.so.6
2172 -rwxr-xr-x 1 root root 2220400 May 6 22:34 /lib/x86_64-linux-gnu/libc.so.6

```

Does that make sense to you?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2024, 12:37pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/18 "2024-07-22T12:37:48Z")

</div>

What’s

```julia
filemode("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")

```

or

```julia
stat("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")

```

? I’m asking this question because `ispath` simply checks the mode of the file:

> <https://github.com/JuliaLang/julia/blob/48d4fd48430af58502699fdf3504b90589df3852/base/stat.jl#L305>

---

<div class="post-metadata">

**Author:** ![engram](https://avatars.discourse-cdn.com/v4/letter/e/94ad74/32.png) [@engram](https://discourse.julialang.org/u/engram)\
**Post date:** [July 22, 2024, 1:43pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/19 "2024-07-22T13:43:41Z")

</div>

that yields:

```julia-repl
julia> stat("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")
StatStruct for "/lib/x86_64-linux-gnu/libcuda.so.535.161.08"
   size: 0 bytes
 device: 0
  inode: 0
   mode: 0o000000 (----------)
  nlink: 0
    uid: 0 (root)
    gid: 0 (root)
   rdev: 0
  blksz: 0
 blocks: 0
  mtime: 1970-01-01T01:00:00+0100 (19926 days ago)
  ctime: 1970-01-01T01:00:00+0100 (19926 days ago)
julia> filemode("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")
0x0000000000000000

```

Checking the permissions of the file results in:

```julia-repl
julia> uperm("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")
0x00

julia> gperm("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")
0x00

julia> operm("/lib/x86_64-linux-gnu/libcuda.so.535.161.08")
0x00

```

in comparison to the specification in the terminal:

```julia
$ ls -sla /lib/x86_64-linux-gnu/libcuda.so.535.161.08 
28696 -rwxr-xr-x 1 root root 29380752 Mar 6 01:04 /lib/x86_64-linux-gnu/libcuda.so.535.161.08

```

That explains why I cannot `dlopen` it, but why does it have different permissions?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2024, 2:30pm UTC](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186/20 "2024-07-22T14:30:29Z")

</div>

> [@engram](#):
>
> but why does it have 0 bytes?

No clue 🙂 But you said you’re running in a sort of virtualised environment, right? Do you know how it was set up exactly? It looks like this file is a bit too virtual.

[Next page](https://discourse.julialang.org/t/cuda-cuda-driver-not-found/117186.md?page=2)