# Fresh CUDA and LuxCUDA error ERROR: could not load symbol "cublasLtMatmulDescCreate":

**URL:** <https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401>\
**Category:** GPU\
**Tags:** gpu, cuda, lux\
**Created:** [May 17, 2024, 3:08pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401 "2024-05-17T15:08:20Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmetivie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmetivie/32/6926_2.png) [@dmetivie](https://discourse.julialang.org/u/dmetivie)\
**Post date:** [May 17, 2024, 3:08pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/1 "2024-05-17T15:08:20Z")

</div>

From [Lux polynomial fitting tutorial](https://lux.csail.mit.edu/v0.5.30/tutorials/beginner/2_PolynomialFitting) I get during the training step

```julia
ERROR: could not load symbol "cublasLtMatmulDescCreate":
The specified procedure could not be found.

```

I did the `Cuda.test()` and got errors

```julia
core/nvml (3) | failed 

```

```julia
versioninfo()
Julia Version 1.10.3
Commit 0b4590a550 (2024-04-30 10:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 28 × 13th Gen Intel(R) Core(TM) i7-13850HX
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, goldmont)
Threads: 10 default, 0 interactive, 5 GC (on 28 virtual cores)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 10

```

```julia
CUDA.versioninfo()
CUDA runtime 12.4, artifact installation
CUDA driver 12.2
NVIDIA driver 538.27.0

CUDA libraries:
- CUBLAS: 12.4.5
- CURAND: 10.3.5
- CUFFT: 11.2.1
- CUSOLVER: 11.6.1
- CUSPARSE: 12.3.1
- CUPTI: 22.0.0
- NVML: 12.0.0+538.27

Julia packages:
- CUDA: 5.3.4
- CUDA_Driver_jll: 0.8.1+0
- CUDA_Runtime_jll: 0.12.1+0

Toolchain:
- Julia: 1.10.3
- LLVM: 15.0.7

1 device:
  0: NVIDIA RTX 3500 Ada Generation Laptop GPU (sm_89, 10.931 GiB / 11.994 GiB available)

```

Should I specify some path to `cublasLtMatmulDescCreate` if yes where should it be?

---

<div class="post-metadata">

**Author:** ![avikpal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avikpal/32/6550_2.png) [@avikpal](https://discourse.julialang.org/u/avikpal)\
**Post date:** [May 17, 2024, 3:54pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/2 "2024-05-17T15:54:44Z")

</div>

@maleadt CuBLASLt should be present in all the new CUDA versions, right?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [May 18, 2024, 7:35am UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/3 "2024-05-18T07:35:03Z")

</div>

Yeah, but we seem to be `ccall`ing `libcublas` instead of `libcublasLt`. Can you open an issue?

---

<div class="post-metadata">

**Author:** ![dmetivie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmetivie/32/6926_2.png) [@dmetivie](https://discourse.julialang.org/u/dmetivie)\
**Post date:** [May 18, 2024, 10:54pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/4 "2024-05-18T22:54:10Z")

</div>

Thanks for looking into it.

Note that this seems to be a Windows only issue since it works on my WSL (however GPU is much slower than CPU while training, even large models where the GPU cost should be negligible).

---

<div class="post-metadata">

**Author:** ![avikpal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avikpal/32/6550_2.png) [@avikpal](https://discourse.julialang.org/u/avikpal)\
**Post date:** [May 18, 2024, 11:12pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/5 "2024-05-18T23:12:13Z")

</div>

can you set `CUDA.allowscalar(false)` and check if any of the dispatches are hitting generic kernels?

---

<div class="post-metadata">

**Author:** ![dmetivie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmetivie/32/6926_2.png) [@dmetivie](https://discourse.julialang.org/u/dmetivie)\
**Post date:** [May 19, 2024, 6:47pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/6 "2024-05-19T18:47:59Z")

</div>

> [@avikpal](#):
>
> `CUDA.allowscalar(false)`

I tested in both Windows and WSL (not sure which you were referring to)  
It does produce the same error on Windows and does not change the timing in WSL.

---

<div class="post-metadata">

**Author:** ![dmetivie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmetivie/32/6926_2.png) [@dmetivie](https://discourse.julialang.org/u/dmetivie)\
**Post date:** [May 19, 2024, 7:09pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/7 "2024-05-19T19:09:31Z")

</div>

My bad for WSL seems indeed faster with GPU when adding layers.

---

<div class="post-metadata">

**Author:** ![dmetivie](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmetivie/32/6926_2.png) [@dmetivie](https://discourse.julialang.org/u/dmetivie)\
**Post date:** [May 21, 2024, 8:09am UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/8 "2024-05-21T08:09:27Z")

</div>

Now while precompiling (on Windows) I got this warning

```julia
│ ┌ Warning: cuBLASLt is not functional on this system. We won't be able to use optimized implementations of certain matmul operations.
```

---

<div class="post-metadata">

**Author:** ![avikpal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avikpal/32/6550_2.png) [@avikpal](https://discourse.julialang.org/u/avikpal)\
**Post date:** [May 21, 2024, 9:06am UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/9 "2024-05-21T09:06:58Z")

</div>

Yes, we try to detect if the current cuBLASLt setting works for the given system, and if it doesn’t, we just use cuBLAS.

---

<div class="post-metadata">

**Author:** ![Alergy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alergy/32/202344_2.png) [@Alergy](https://discourse.julialang.org/u/Alergy)\
**Post date:** [May 22, 2024, 9:52am UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/10 "2024-05-22T09:52:55Z")

</div>

I got the same error using the LuxCUDA.jl package on Windows for a tutorial on [NeuralODEs](https://docs.sciml.ai/DiffEqFlux/dev/examples/GPUs/). It occurred just when I try to solve the problem on the GPU.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [May 22, 2024, 12:07pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/11 "2024-05-22T12:07:34Z")

</div>

What’s the reason for that? CUBLASLt should work fine on Windows.

---

<div class="post-metadata">

**Author:** ![avikpal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avikpal/32/6550_2.png) [@avikpal](https://discourse.julialang.org/u/avikpal)\
**Post date:** [May 23, 2024, 3:05am UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/12 "2024-05-23T03:05:55Z")

</div>

It is this error:

```julia
ERROR: could not load symbol "cublasLtMatmulDescCreate":
The specified procedure could not be found.

```

It is probably stemming from the ccall into `libcublas` instead of `libcublaslt` (the problem you mentioned). I added a check in ` __init__ ()` to verify if a simple matmul works via cuBLASLt and if not, don’t try to use that at runtime.

---

<div class="post-metadata">

**Author:** ![avikpal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avikpal/32/6550_2.png) [@avikpal](https://discourse.julialang.org/u/avikpal)\
**Post date:** [May 23, 2024, 3:07am UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/13 "2024-05-23T03:07:25Z")

</div>

Are you getting the error on the latest releases? Or are you simply getting a warning, in which case you should ignore it?

---

<div class="post-metadata">

**Author:** ![Alergy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alergy/32/202344_2.png) [@Alergy](https://discourse.julialang.org/u/Alergy)\
**Post date:** [May 31, 2024, 12:12am UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/14 "2024-05-31T00:12:51Z")

</div>

I think I was getting that on the latest releases. I remember that it was not just a warning, the operations on the GPU did not work.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [May 31, 2024, 4:41pm UTC](https://discourse.julialang.org/t/fresh-cuda-and-luxcuda-error-error-could-not-load-symbol-cublasltmatmuldesccreate/114401/15 "2024-05-31T16:41:22Z")

</div>

This should be fixed on the latest version of CUDA.jl.