CUDA: CUDA driver not found

giordano · July 22, 2024, 2:50pm

To elaborate, this is the source code of the C function used to stat files:

JuliaLang/julia/blob/e621c74d52e095d740cede89e4e9da82d4f737a9/src/sys.c#L112-L124


      
          JL_DLLEXPORT int32_t jl_stat(const char *path, char *statbuf) JL_NOTSAFEPOINT
          {
              uv_fs_t req;
              int ret;
          
              // Ideally one would use the statbuf for the storage in req, but
              // it's not clear that this is possible using libuv
              ret = uv_fs_stat(unused_uv_loop_arg, &req, path, NULL);
              if (ret == 0)
                  memcpy(statbuf, req.ptr, sizeof(uv_stat_t));
              uv_fs_req_cleanup(&req);
              return ret;
          }

It’s basically using libuv’s uv_fs_stat.

engram · July 23, 2024, 9:40am

haha indeed
Looking into the virtualisation I could figure it out. So in order to enable users access to different software from different machines they setup modulefiles through which the environment can be changed on-the-fly e.g. add an entry to PATH. Thus users don’t have to setup software on their own and can switch between versions.

In my case module load julia added symlink named julia to the PATH variable. That symlink points towards a singularity wrapper that runs a singularity container in which julia is installed. Of course within the container the “new” driver library cannot be found if not bound/mounted. So adding

export SINGULARITY_BIND="${SINGULARITY_BIND},/lib/x86_64-linux-gnu/libcuda.so.535.161.08:/.singularity.d/libs/libcuda.so"

to my .bashrc mounts the driver, enables julia to find libcuda.so within the container. Consequenty using Base.Libc.Libdl; dlopen("libcuda.so") works, and also CUDA.functional() returns true.

In Hindsight I should have looked into that sooner.
Thank both of you again for your time and advice!

giordano · July 23, 2024, 10:04am

When you were running the C programs to try and reproduce the issue you were outside of the Singularity container, which explains why they were working fine?

engram · July 23, 2024, 11:46am

exactly! Only when using julia the container is run and julia is executed from within.

carstenbauer · July 24, 2024, 5:08am