# Compiling CUDA code with BinaryBuilder

**URL:** <https://discourse.julialang.org/t/compiling-cuda-code-with-binarybuilder/109638>\
**Category:** Package Management\
**Tags:** question, binarybuilder\
**Created:** [February 2, 2024, 6:01pm UTC](https://discourse.julialang.org/t/compiling-cuda-code-with-binarybuilder/109638 "2024-02-02T18:01:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![PaulVirally](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulvirally/32/29911_2.png) [@PaulVirally](https://discourse.julialang.org/u/PaulVirally)\
**Post date:** [February 2, 2024, 6:01pm UTC](https://discourse.julialang.org/t/compiling-cuda-code-with-binarybuilder/109638/1 "2024-02-02T18:01:05Z")

</div>

I apologize in advance for my question, because I know it is not very specific, but at this point I do not know where else to go for help.

I’ve been trying to compile some CUDA code using BinaryBuilder, but I haven’t been able to succeed. I really don’t think I need anything fancy to compile [my code](https://github.com/PaulVirally/VkFFTCUDALib), but I do need `nvcc`, and this is what I have been struggling with. Here is what I have tried so far:

## Trying to load CUDA as a dependency

I tried copying some `build_tarballs.jl` like [NCCL’s `build_tarballs.jl`](https://github.com/JuliaPackaging/Yggdrasil/blob/82d1909ed208362fa2ea68753fdbf56d903584c2/N/NCCL/build_tarballs.jl), but I can’t even get that to run. When I try to NCCL’s script with `julia build_tarballs.jl --verbose --debug` (only a locally cloned version of Yggdrasil) I get an error:

> `ERROR: LoadError: KeyError: key v"11.4.4" not found`

where 11.4.4 corresponds to the CUDA version that is trying to be installed. The problem here seems to be that `CUDA.required_dependencies` (from [`platforms/cuda.jl`](https://github.com/JuliaPackaging/Yggdrasil/blob/master/platforms/cuda.jl)) outputs dependencies that can’t be found. Here is a concrete example:

For the target `x86_64-linux-gnu-cuda+11.4`, that is, `platform = Platform("x86_64", "linux"; libc="glibc", cuda="11.4")`, `CUDA.required_dependencies(platform)` returns the following two dependencies:

- `BuildDependency(PackageSpec(name="CUDA_SDK_jll", version=v"11.4.4"))`
- `BuildDependency(PackageSpec(name="CUDA_Runtime_jll"))`

But then when I run `build_tarballs` with these dependencies, I get the keyerror 11.4.4 not found error. So then I figured I would put the exact version string that I found in the `JuliaBinaryWrappers` repo on github. At this point, I have the following:

```julia
platforms = [Platform("x86_64", "linux"; libc="glibc", cuda="12.3")]
dependencies = [
	HostBuildDependency(PackageSpec(; name="CMake_jll", version=v"3.28.1+0")), # We need cmake >= 3.18, but by default it is 3.17.2
    BuildDependency(PackageSpec(name="CUDA_SDK_jll", version=v"12.3.2+0")),
    RuntimeDependency(PackageSpec(name="CUDA_Runtime_jll")),
]

```

But now when I get dropped into the sandbox with these dependencies, `nvcc` doesn’t work. I tried compiling a hello world, but I get the following error:

```julia-auto
/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -lcudadevrt
/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -lcudart_static

```

This led me to [an issue for NCCL’s build](https://github.com/JuliaPackaging/Yggdrasil/pull/7794#issuecomment-1852892110), but nothing I found there seemed to solve my problems. I also thought that since the error is talking about `cudart_static`, that I might try and add `CUDA_SDK_static_jll` as a dependency, but I still have the same error from `ld` when running `nvcc`. I also tried adding `CUDA_full_jll`, but that didn’t help either. I feel like I am very close with this setup, but I cannot get `cudadevrt` or `cudart_static` to be found. (I of course did try to add `$prefix/cuda/lib` to `LD_LIBRARY_PATH`, but that did not do anything either).

## Trying to manually install CUDA in the sandbox

Then I figured I might as well just install CUDA from the sandbox environment. I found [this recipe](https://github.com/JuliaPackaging/Yggdrasil/blob/8bd138b9e3c8122105a865bf94926834dc6b37a9/X/XRTServer/build_tarballs.jl) which I started copying, but they have install scripts for CUDA 10 which I don’t think supports the compute architecture sm\_80 which I need. I tried to use more up-to-date files:

```julia
sources = [
    FileSource("https://us.download.nvidia.com/XFree86/Linux-x86_64/550.40.07/NVIDIA-Linux-x86_64-550.40.07.run", "298936c727b7eefed95bb87eb8d24cfeef1f35fecac864d98e2694d37749a4ad"),
    FileSource("https://developer.download.nvidia.com/compute/cuda/12.3.2/local_installers/cuda_12.3.2_545.23.08_linux.run", "24b2afc9f770d8cf43d6fa7adc2ebfd47c4084db01bdda1ce3ce0a4d493ba65b"),
]

```

but I’ve honestly had a hard time trying to install CUDA this way within the sandbox. This is where I am stuck currently.

Is there a better way of doing this? Is there a certain set of dependencies that lets me have access to `nvcc`, or do I have to build CUDA myself in the sandbox?

Thank you!

---

<div class="post-metadata">

**Author:** ![SteffenPL](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/steffenpl/32/206270_2.png) [@SteffenPL](https://discourse.julialang.org/u/SteffenPL)\
**Post date:** [February 3, 2024, 2:17am UTC](https://discourse.julialang.org/t/compiling-cuda-code-with-binarybuilder/109638/2 "2024-02-03T02:17:45Z")

</div>

I’m not an expert in this. As far as I can see the `CUDA_Runtime` provides `ptxas` which is a compiler downstream of `nvcc`. Not sure if that helps, but as far as I can tell CUDA.jl uses that level and hence there was. o need for `nvcc`.

In the existing build scripts `nvcc` seems to be at a specific path, that’s maybe why it wasn’t available in the sandbox, see [line in build script](https://github.com/JuliaPackaging/Yggdrasil/blob/7eab9ce14fd0581b2b3443ca7dd2282102b294f0/C/CUDA/CUDA_Runtime/build_11.6.jl#L61). Maybe you could suggest a PR to add `nvcc` as a file product.

Just as a side question, looking at the code it seems using CUDA.jl would be easier to maintain. (But I don’t know if there are bits that don’t work. Obviously you know what you do.)

---

<div class="post-metadata">

**Author:** ![jonas-kr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonas-kr/32/1810_2.png) [@jonas-kr](https://discourse.julialang.org/u/jonas-kr)\
**Post date:** [August 20, 2024, 5:22pm UTC](https://discourse.julialang.org/t/compiling-cuda-code-with-binarybuilder/109638/3 "2024-08-20T17:22:42Z")

</div>

Although this topic is quite old, I have just stumbled across the same issue. Fortunately, I have been able to fix the error as described in [this PR to Yggdrasil](https://github.com/JuliaPackaging/Yggdrasil/pull/9284).

---

<div class="post-metadata">

**Author:** ![PaulVirally](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulvirally/32/29911_2.png) [@PaulVirally](https://discourse.julialang.org/u/PaulVirally)\
**Post date:** [August 25, 2024, 3:22am UTC](https://discourse.julialang.org/t/compiling-cuda-code-with-binarybuilder/109638/4 "2024-08-25T03:22:02Z")

</div>

Thank you for this! It turns out the solution [was to use Julia 1.7](https://github.com/JuliaPackaging/Yggdrasil/pull/9284#issuecomment-2301523466)