# CUDA test random failure

**URL:** <https://discourse.julialang.org/t/cuda-test-random-failure/92399>\
**Category:** GPU\
**Created:** [January 2, 2023, 12:20am UTC](https://discourse.julialang.org/t/cuda-test-random-failure/92399 "2023-01-02T00:20:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jinfreedom](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@jinfreedom](https://discourse.julialang.org/u/jinfreedom)\
**Post date:** [January 2, 2023, 12:20am UTC](https://discourse.julialang.org/t/cuda-test-random-failure/92399/1 "2023-01-02T00:20:44Z")

</div>

Hi, I am new to Julia gpu computation. I recently installed `CUDA.jl`, with the following status:

```julia
(julia) pkg> st
Status `~/julia/Project.toml`
  [6e4b80f9] BenchmarkTools v1.3.2
  [052768ef] CUDA v3.12.0
  [7a1cc6ca] FFTW v1.5.0
  [f67ccb44] HDF5 v0.16.13
  [033835bb] JLD2 v0.4.29
  [91a5bcdd] Plots v1.38.0

```

Upon testing with `]test CUDA` I received the following info along with some failures of various errors. Following suggestions by the test message, I used `--threads n` with different n’s, but regardless I randomly got some `worker # terminated` message. Can someone help me with these errors? So far I tried several times and have not been able to finish the test even once. Part of the screen output is shown below, with all relevant version info. Thanks in advance.

```julia
...
  [13072b0f] AxisAlgorithms v1.0.1
⌅ [ab4f0b2a] BFloat16s v0.2.0
  [fa961155] CEnum v0.4.2
...
 [46192b85] GPUArraysCore v0.1.2
⌅ [61eb1bfa] GPUCompiler v0.16.7
  [a98d9a8b] Interpolations v0.14.7
...
        Info Packages marked with ⌅ have new versions available but cannot be upgraded.
     Testing Running tests...
┌ Info: System information:
│ CUDA toolkit 11.7, artifact installation
│ NVIDIA driver 527.56.0, for CUDA 12.0
│ CUDA driver 12.0
│
│ Libraries:
│ - CUBLAS: 11.10.1
│ - CURAND: 10.2.10
│ - CUFFT: 10.7.2
│ - CUSOLVER: 11.3.5
│ - CUSPARSE: 11.7.3
│ - CUPTI: 17.0.0
│ - NVML: 12.0.0+525.65
│ - CUDNN: 8.30.2 (for CUDA 11.5.0)
│ - CUTENSOR: 1.4.0 (for CUDA 11.5.0)
│
│ Toolchain:
│ - Julia: 1.8.1
│ - LLVM: 13.0.1
│ - PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0, 7.1, 7.2
│ - Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86
│
│ 1 device:
└ 0: NVIDIA GeForce RTX 3060 Laptop GPU (sm_86, 4.956 GiB / 6.000 GiB available)
[ Info: Testing using 1 device(s): 0. NVIDIA GeForce RTX 3060 Laptop GPU (UUID f780bc3c-a9f0-9c3d-e723-c01de3450fd7)
...
                                                  | | ---------------- GPU ---------------- | ---------------- CPU ---------------- |
Test (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
initialization (2) | 6.50 | 0.00 | 0.0 | 0.00 | N/A | 0.02 | 0.3 | 122.10 | 891.5
...
Worker 5 terminated.
gpuarrays/linalg/mul!/matrix-matrix (5) | failed at 2023-01-01T18:51:32.221
Unhandled Task ERROR: EOFError: read end of file
...
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
...
Worker 4 terminated.
gpuarrays/linalg/norm (4) | failed at 2023-01-01T18:57:34.022
Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
gpuarrays/base (3) | 269.04 | 0.07 | 0.0 | 8.90 | N/A | 2.51 | 0.9 | 5865.70 | 2267.16 |
...
broadcast (7) | 23.83 | 0.00 | 0.0 | 0.00 | N/A | 1.47 | 6.2 | 3407.76 | 1668.28 |
codegen (7) | failed at 2023-01-01T19:06:18.211
array (8) | 137.78 | 0.14 | 0.1 | 1261.61 | N/A | 11.54 | 8.4 | 24142.80 | 2255.11 |
      From worker 8:
      From worker 8: signal (11): Segmentation fault
...
      From worker 8: unknown function (ip: 0x7f86dc6406e5)
      From worker 8: unknown function (ip: 0x7f8669b71768)
      From worker 8: unknown function (ip: 0x7f8668f62343)
...
Worker 8 terminated.
cudadrv (8) | failed at 2023-01-01T19:06:36.235
Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
Worker 3 terminated.
gpuarrays/broadcasting (3) | failed at 2023-01-01T19:16:14.934
Unhandled Task ERROR: EOFError: read end of file
...
```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 2, 2023, 11:38am UTC](https://discourse.julialang.org/t/cuda-test-random-failure/92399/2 "2023-01-02T11:38:48Z")

</div>

> [@jinfreedom](#):
>
> ```julia
> From worker 8: signal (11): Segmentation fault
> ...
> From worker 8: unknown function (ip: 0x7f86dc6406e5)
> From worker 8: unknown function (ip: 0x7f8669b71768)
> From worker 8: unknown function (ip: 0x7f8668f62343)
> 
> ```

This is the crucial part, but you left out the actual interesting part of the stack trace. Could you post the other lines? Does it point to a specific location?

> [@jinfreedom](#):
>
> `│ CUDA driver 12.0`

This is a very new version, and CUDA.jl may not be fully compatible yet. Try the master branch, there is at least one segfault I fixed: [Change invalid code test for CUDA 12 compatibility. · JuliaGPU/CUDA.jl@c40daf8 · GitHub](https://github.com/JuliaGPU/CUDA.jl/commit/c40daf89bb3154a9949a785b02db07bef223be16). If this is the error you were running into, know that it doesn’t affect any realistic use case, so you should be fine just using CUDA.jl in your environment.

---

<div class="post-metadata">

**Author:** ![jinfreedom](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@jinfreedom](https://discourse.julialang.org/u/jinfreedom)\
**Post date:** [January 4, 2023, 2:46pm UTC](https://discourse.julialang.org/t/cuda-test-random-failure/92399/3 "2023-01-04T14:46:06Z")

</div>

> [@maleadt](#):
>
> This is the crucial part, but you left out the actual interesting part of the stack trace. Could you post the other lines? Does it point to a specific location?

The output is too long to copy by hand, so I re-tested with the `master` branch as suggested and redirected outputs to `out.log` as follows

```julia
julia --threads 4 test.jl &> out.log

```

where contents of `test.jl` is as follows

```julia
using Pkg
Pkg.activate(".")
Pkg.status()
Pkg.test("CUDA")

```

Unfortunately there are still plenty of errors, and color coding of the screen outputs is lost. Half of the output is attached. Apologies for the length.  
Are these errors not important? If they are, is there a way to “retrofit” the `CUDA 12.0` toolkit back to a compatible version? So far my play with the `CUDA` package seems to work. I’m using wsl2 Ubuntu 20.04.5 LTS

```julia
    Activating project at `~/julia`
     Testing CUDA
      Status `/tmp/jl_bUo2zf/Project.toml`
  [79e6a3ab] Adapt v3.4.0
  [ab4f0b2a] BFloat16s v0.4.2
  [052768ef] CUDA v4.0.0 `https://github.com/JuliaGPU/CUDA.jl.git#master`
  [864edb3b] DataStructures v0.18.13
  [7a1cc6ca] FFTW v1.5.0
  [0c68f7d7] GPUArrays v8.5.0
  [a98d9a8b] Interpolations v0.14.7
  [872c559c] NNlib v0.8.13
  [276daf66] SpecialFunctions v2.1.7
  [a759f4b9] TimerOutputs v0.5.22
  [76a88914] CUDA_Runtime_jll v0.2.3+2
  [ade2ca70] Dates `@stdlib/Dates`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_bUo2zf/Manifest.toml`
  [621f4979] AbstractFFTs v1.2.1
  [79e6a3ab] Adapt v3.4.0
  [13072b0f] AxisAlgorithms v1.0.1
  [ab4f0b2a] BFloat16s v0.4.2
  [fa961155] CEnum v0.4.2
  [052768ef] CUDA v4.0.0 `https://github.com/JuliaGPU/CUDA.jl.git#master`
  [1af6417a] CUDA_Runtime_Discovery v0.1.1
  [d360d2e6] ChainRulesCore v1.15.6
  [9e997f8a] ChangesOfVariables v0.1.4
  [34da2185] Compat v4.5.0
  [864edb3b] DataStructures v0.18.13
  [ffbed154] DocStringExtensions v0.9.3
  [e2ba6199] ExprTools v0.1.8
  [7a1cc6ca] FFTW v1.5.0
  [0c68f7d7] GPUArrays v8.5.0
  [46192b85] GPUArraysCore v0.1.2
  [61eb1bfa] GPUCompiler v0.17.0
  [a98d9a8b] Interpolations v0.14.7
  [3587e190] InverseFunctions v0.1.8
  [92d709cd] IrrationalConstants v0.1.1
  [692b3bcd] JLLWrappers v1.4.1
  [929cbde3] LLVM v4.14.1
  [2ab3a3ac] LogExpFunctions v0.3.19
  [872c559c] NNlib v0.8.13
  [6fe1bfb0] OffsetArrays v1.12.8
  [bac558e1] OrderedCollections v1.4.1
  [21216c6a] Preferences v1.3.0
  [74087812] Random123 v1.6.0
  [e6cf234a] RandomNumbers v1.5.3
  [c84ed2f1] Ratios v0.4.3
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [276daf66] SpecialFunctions v2.1.7
  [90137ffa] StaticArrays v1.5.12
  [1e83bf80] StaticArraysCore v1.4.0
  [a759f4b9] TimerOutputs v0.5.22
  [efce3f68] WoodburyMatrices v0.5.5
  [4ee394cb] CUDA_Driver_jll v0.2.0+0
  [76a88914] CUDA_Runtime_jll v0.2.3+2
  [f5851436] FFTW_jll v3.3.10+0
  [1d5cc7b8] IntelOpenMP_jll v2018.0.3+2
  [dad2f222] LLVMExtra_jll v0.0.16+0
  [856f044c] MKL_jll v2022.2.0+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [4af54fe1] LazyArtifacts `@stdlib/LazyArtifacts`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.8.0 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [1a1011a3] SharedArrays `@stdlib/SharedArrays`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [fa267f1f] TOML v1.0.0 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.0 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll v7.84.0+0 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.10.2+0 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.28.0+0 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2022.2.1 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.20+0 `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll v0.8.1+0 `@stdlib/OpenLibm_jll`
  [83775a58] Zlib_jll v1.2.12+3 `@stdlib/Zlib_jll`
  [8e850b90] libblastrampoline_jll v5.1.1+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.48.0+0 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
     Testing Running tests...
                                                  | | ---------------- GPU ---------------- | ---------------- CPU ---------------- |
Test (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
initialization (2) | 9.97 | 0.00 | 0.0 | 0.00 | N/A | 0.01 | 0.1 | 97.03 | 1209.04 |
gpuarrays/indexing scalar (3) | 34.00 | 0.04 | 0.1 | 0.01 | N/A | 1.44 | 4.2 | 4906.21 | 1212.18 |
gpuarrays/math/power (2) | 79.78 | 0.00 | 0.0 | 0.01 | N/A | 7.10 | 8.9 | 14302.66 | 2061.98 |
gpuarrays/linalg/mul!/vector-matrix (3) | 92.43 | 0.01 | 0.0 | 0.02 | N/A | 5.47 | 5.9 | 14498.98 | 2192.13 |
gpuarrays/interface (3) | 5.51 | 0.00 | 0.0 | 0.00 | N/A | 0.32 | 5.8 | 761.85 | 2192.13 |
gpuarrays/indexing multidimensional (2) | 49.48 | 0.00 | 0.0 | 1.21 | N/A | 2.77 | 5.6 | 7719.91 | 2061.98 |
gpuarrays/linalg (5) | 143.29 | 0.07 | 0.0 | 11.72 | N/A | 7.63 | 5.3 | 22060.00 | 2457.75 |
gpuarrays/reductions/reducedim! (4) | 145.92 | 0.04 | 0.0 | 1.03 | N/A | 13.05 | 8.9 | 26430.49 | 1226.55 |
gpuarrays/uniformscaling (5) | 12.69 | 0.00 | 0.0 | 0.01 | N/A | 0.56 | 4.4 | 1509.86 | 2457.75 |
gpuarrays/reductions/any all count (3) | 26.58 | 0.00 | 0.0 | 0.00 | N/A | 2.73 | 10.3 | 5366.74 | 2192.13 |
gpuarrays/math/intrinsics (5) | 4.75 | 0.00 | 0.0 | 0.00 | N/A | 0.25 | 5.3 | 665.09 | 2457.75 |
gpuarrays/statistics (5) | failed at 2023-01-04T02:03:35.980
gpuarrays/linalg/mul!/matrix-matrix (4) | 809.35 | 1.01 | 0.1 | 0.12 | N/A | 11.21 | 1.4 | 23473.99 | 1895.87 |
gpuarrays/constructors (4) | 46.32 | 0.01 | 0.0 | 0.08 | N/A | 2.35 | 5.1 | 6033.94 | 2382.25 |
gpuarrays/random (4) | 32.03 | 0.00 | 0.0 | 0.03 | N/A | 1.90 | 5.9 | 4588.74 | 2382.25 |
gpuarrays/linalg/norm (3) | failed at 2023-01-04T02:08:00.870
gpuarrays/base (4) | 124.02 | 1.99 | 1.6 | 8.90 | N/A | 3.17 | 2.6 | 6723.08 | 2382.25 |
gpuarrays/reductions/minimum maximum extrema (2) | 1082.80 | 0.62 | 0.1 | 2.19 | N/A | 34.26 | 3.2 | 61498.86 | 2202.61 |
gpuarrays/reductions/mapreduce (6) | 337.42 | 0.23 | 0.1 | 1.81 | N/A | 20.98 | 6.2 | 44044.26 | 1314.33 |
gpuarrays/reductions/== isequal (7) | 115.35 | 0.09 | 0.1 | 1.07 | N/A | 9.27 | 8.0 | 20251.40 | 1210.26 |
gpuarrays/reductions/reduce (6) | 40.15 | 0.01 | 0.0 | 1.21 | N/A | 1.10 | 2.8 | 2833.10 | 1516.86 |
apiutils (6) | 0.15 | 0.00 | 0.0 | 0.00 | N/A | 0.00 | 0.0 | 0.80 | 1516.88 |
gpuarrays/reductions/mapreducedim! (2) | failed at 2023-01-04T02:15:43.180
broadcast (8) | 37.63 | 0.04 | 0.1 | 0.00 | N/A | 2.38 | 6.3 | 5834.69 | 1203.00 |
codegen (8) | failed at 2023-01-04T02:16:47.645
gpuarrays/broadcasting (4) | failed at 2023-01-04T02:29:47.941
array (6) | 1191.20 | 0.84 | 0.1 | 1264.82 | N/A | 12.89 | 1.1 | 22975.35 | 3057.52 |
cudadrv (10) | 20.93 | 0.09 | 0.4 | 0.00 | N/A | 0.68 | 3.3 | 2345.37 | 1204.49 |
curand (10) | 0.46 | 0.00 | 0.0 | 0.00 | N/A | 0.01 | 2.0 | 35.80 | 1204.49 |
cufft (6) | 40.04 | 0.01 | 0.0 | 205.31 | N/A | 2.59 | 6.5 | 4279.44 | 3136.33 |
examples (6) | failed at 2023-01-04T02:31:34.058
cusparse (10) | 97.95 | 0.12 | 0.1 | 9.47 | N/A | 5.22 | 5.3 | 10453.82 | 1967.34 |
      From worker 10:	WARNING: Method definition #5122#kernel(Any) in module Main at /home/sunjin/.julia/packages/CUDA/ifpPv/test/execution.jl:315 overwritten at /home/sunjin/.julia/packages/CUDA/ifpPv/test/execution.jl:323.
cublas (9) | failed at 2023-01-04T02:32:57.914
execution (10) | failed at 2023-01-04T02:33:11.848
gpuarrays/reductions/sum prod (7) | 1378.23 | 48.74 | 3.5 | 3.24 | N/A | 30.53 | 2.2 | 53438.22 | 2154.23 |
iterator (12) | 4.25 | 0.04 | 1.0 | 1.93 | N/A | 0.19 | 4.4 | 664.93 | 1201.90 |
nvtx (12) | 0.22 | 0.00 | 0.0 | 0.00 | N/A | 0.00 | 0.0 | 22.61 | 1201.90 |
pointer (12) | 0.29 | 0.00 | 0.0 | 0.00 | N/A | 0.00 | 0.0 | 13.19 | 1201.90 |
exceptions (11) | 86.29 | 0.00 | 0.0 | 0.00 | N/A | 0.00 | 0.0 | 34.57 | 1202.88 |
pool (12) | 1.43 | 0.00 | 0.0 | 0.00 | N/A | 0.30 | 21.0 | 204.39 | 1201.90 |
nvml (13) | 0.54 | 0.00 | 0.0 | 0.00 | N/A | 0.00 | 0.0 | 59.84 | 1198.10 |
linalg (7) | 46.79 | 0.00 | 0.0 | 9.03 | N/A | 3.55 | 7.6 | 7630.12 | 3462.21 |
random (11) | 45.98 | 0.05 | 0.1 | 256.58 | N/A | 2.70 | 5.9 | 6763.16 | 1202.88 |
utils (11) | 0.86 | 0.00 | 0.0 | 0.00 | N/A | 0.02 | 2.0 | 85.98 | 1202.88 |
threading (7) | 5.63 | 0.00 | 0.1 | 10.94 | N/A | 0.44 | 7.9 | 564.09 | 3462.27 |
cusolver/multigpu (7) | 18.82 | 0.00 | 0.0 | 545.89 | N/A | 0.82 | 4.3 | 1914.40 | 3491.86 |
texture (13) | 65.86 | 0.04 | 0.1 | 0.09 | N/A | 4.19 | 6.4 | 10976.77 | 1198.10 |
cusolver/sparse (7) | failed at 2023-01-04T02:34:38.172
cusparse/conversions (14) | 31.69 | 0.12 | 0.4 | 1.43 | N/A | 1.90 | 6.0 | 4384.70 | 1204.48 |
cusparse/device (14) | 0.33 | 0.00 | 0.1 | 0.01 | N/A | 0.01 | 2.9 | 7.49 | 1204.48 |
cusparse/broadcast (13) | 60.58 | 0.01 | 0.0 | 0.02 | N/A | 3.78 | 6.2 | 9708.01 | 1237.14 |
sorting (12) | 171.13 | 0.01 | 0.0 | 543.84 | N/A | 8.18 | 4.8 | 28184.76 | 3912.35 |
cusparse/generic (14) | 55.49 | 0.05 | 0.1 | 6.52 | N/A | 2.64 | 4.7 | 6108.44 | 1211.70 |
device/array (14) | 8.73 | 0.00 | 0.0 | 0.00 | N/A | 0.53 | 6.1 | 1371.40 | 1211.71 |
cusparse/interfaces (13) | 87.57 | 0.09 | 0.1 | 10.75 | N/A | 3.85 | 4.4 | 9440.00 | 1420.38 |
device/intrinsics (14) | 39.17 | 0.00 | 0.0 | 0.00 | N/A | 2.42 | 6.2 | 6385.59 | 1238.05 |
device/ldg (13) | 10.24 | 0.00 | 0.0 | 0.00 | N/A | 0.68 | 6.6 | 1569.36 | 1420.38 |
cusparse/linalg (12) | failed at 2023-01-04T02:37:18.497
cusolver/dense (11) | 193.07 | 0.13 | 0.1 | 2519.16 | N/A | 13.88 | 7.2 | 29851.59 | 2153.11 |
device/intrinsics/memory (11) | 24.21 | 0.00 | 0.0 | 0.02 | N/A | 1.58 | 6.5 | 3829.37 | 2361.20 |
device/random (14) | 48.45 | 0.00 | 0.0 | 0.17 | N/A | 2.81 | 5.8 | 7153.73 | 1238.09 |
device/intrinsics/output (11) | 25.76 | 0.00 | 0.0 | 0.00 | N/A | 1.86 | 7.2 | 4099.07 | 2361.20 |
device/intrinsics/atomics (13) | 78.17 | 0.00 | 0.0 | 0.00 | N/A | 5.07 | 6.5 | 12078.32 | 1508.60 |
device/intrinsics/math (15) | 70.00 | 0.10 | 0.1 | 0.00 | N/A | 4.00 | 5.7 | 10741.26 | 1336.75 |
device/intrinsics/wmma (14) | 89.49 | 0.01 | 0.0 | 0.63 | N/A | 5.57 | 6.2 | 14501.88 | 1808.08 |
Testing finished in 50 minutes, 55 seconds, 583 milliseconds
gpuarrays/statistics: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(5)
gpuarrays/linalg/norm: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(3)
gpuarrays/reductions/mapreducedim!: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(2)
Worker 8 failed running test codegen:
Some tests did not pass: 13 passed, 0 failed, 3 errored, 0 broken.
codegen: Error During Test at /home/sunjin/.julia/packages/CUDA/ifpPv/test/codegen.jl:170
  Test threw exception
  Expression: CUDA.code_sass(devnull, valid_kernel, Tuple{}) == nothing
  CUPTIError: CUPTI is unable to initialize its connection to the CUDA driver (code 15, CUPTI_ERROR_NOT_INITIALIZED)
  Stacktrace:
    [1] (::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{var"#valid_kernel#71", Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}})()
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:76
    [2] lock(f::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{var"#valid_kernel#71", Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}}, l::ReentrantLock)
      @ Base ./lock.jl:185
    [3] #code_sass#228
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:69 [inlined]
    [4] code_sass(io::Base.DevNull, func::Any, types::Any, kernel::Bool; verbose::Bool, always_inline::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:51
    [5] code_sass (repeats 2 times)
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:45 [inlined]
    [6] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:464 [inlined]
    [7] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:170 [inlined]
    [8] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
    [9] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:167 [inlined]
   [10] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [11] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:166
codegen: Error During Test at /home/sunjin/.julia/packages/CUDA/ifpPv/test/codegen.jl:174
  Got exception outside of a @test
  CUPTIError: CUPTI is unable to initialize its connection to the CUDA driver (code 15, CUPTI_ERROR_NOT_INITIALIZED)
  Stacktrace:
    [1] (::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(kernel_341), Tuple{Ptr{Int64}}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}})()
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:76
    [2] lock(f::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(kernel_341), Tuple{Ptr{Int64}}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}}, l::ReentrantLock)
      @ Base ./lock.jl:185
    [3] #code_sass#228
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:69 [inlined]
    [4] code_sass(io::Base.DevNull, func::Any, types::Any, kernel::Bool; verbose::Bool, always_inline::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:51
    [5] code_sass (repeats 2 times)
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:45 [inlined]
    [6] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:179 [inlined]
    [7] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
    [8] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:175 [inlined]
    [9] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [10] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:166
   [11] include
      @ ./client.jl:476 [inlined]
   [12] #12
      @ ~/.julia/packages/CUDA/ifpPv/test/runtests.jl:101 [inlined]
   [13] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [14] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [15] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [16] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/utilities.jl:25 [inlined]
   [17] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/pool.jl:573 [inlined]
   [18] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:54
   [19] eval
      @ ./boot.jl:368 [inlined]
   [20] runtests(f::Function, name::String, time_source::Symbol, snoop::Nothing)
      @ Main ~/.julia/packages/CUDA/ifpPv/test/setup.jl:66
   [21] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./essentials.jl:729
   [22] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:726
   [23] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285
   [24] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
   [25] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285 [inlined]
   [26] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:484
codegen: Error During Test at /home/sunjin/.julia/packages/CUDA/ifpPv/test/codegen.jl:182
  Got exception outside of a @test
  CUPTIError: CUPTI doesn't allow multiple callback subscribers. Only a single subscriber can be registered at a time. (code 39, CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED)
  Stacktrace:
    [1] (::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{var"#kernel#73", Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}})()
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:76
    [2] lock(f::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{var"#kernel#73", Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}}, l::ReentrantLock)
      @ Base ./lock.jl:185
    [3] #code_sass#228
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:69 [inlined]
    [4] code_sass(io::Base.DevNull, func::Any, types::Any, kernel::Bool; verbose::Bool, always_inline::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:51
    [5] code_sass (repeats 2 times)
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:45 [inlined]
    [6] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:185 [inlined]
    [7] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
    [8] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:183 [inlined]
    [9] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [10] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:166
   [11] include
      @ ./client.jl:476 [inlined]
   [12] #12
      @ ~/.julia/packages/CUDA/ifpPv/test/runtests.jl:101 [inlined]
   [13] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [14] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [15] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [16] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/utilities.jl:25 [inlined]
   [17] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/pool.jl:573 [inlined]
   [18] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:54
   [19] eval
      @ ./boot.jl:368 [inlined]
   [20] runtests(f::Function, name::String, time_source::Symbol, snoop::Nothing)
      @ Main ~/.julia/packages/CUDA/ifpPv/test/setup.jl:66
   [21] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./essentials.jl:729
   [22] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:726
   [23] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285
   [24] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
   [25] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285 [inlined]
   [26] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:484
gpuarrays/broadcasting: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(4)
examples: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(6)
Worker 9 failed running test cublas:
Some tests did not pass: 2180 passed, 2 failed, 0 errored, 0 broken.
cublas: Test Failed at /home/sunjin/.julia/packages/CUDA/ifpPv/test/cublas.jl:1031
  Expression: C ≈ Array(dA)
   Evaluated: ComplexF32[0.22159632f0 + 0.2527002f0im 0.7149214f0 - 0.07908373f0im … 1.584436f0 - 1.0320874f0im 0.88353616f0 + 0.11684081f0im; 0.8638178f0 + 0.19264713f0im 0.8187425f0 - 0.2023008f0im … 1.1051666f0 - 1.1044672f0im 1.1477678f0 + 0.30134824f0im; … ; 0.36396125f0 + 0.2685322f0im 0.53985554f0 + 0.5504699f0im … 1.3564304f0 - 1.298666f0im 0.7817802f0 + 0.5078405f0im; 0.46989524f0 + 0.8126974f0im 1.1645075f0 - 0.007235131f0im … 0.8363035f0 - 1.4039177f0im 0.9743213f0 + 0.30272323f0im] ≈ ComplexF32[-0.101180695f0 + 0.32050708f0im 0.21519771f0 + 0.6863358f0im … -1.32294f0 + 1.3511004f0im 0.021058274f0 + 0.89097947f0im; 0.283161f0 + 0.83851886f0im 0.35609248f0 + 0.76450187f0im … -0.8393788f0 + 1.3178332f0im -0.12041329f0 + 1.1805433f0im; … ; -0.040902115f0 + 0.45044902f0im -0.43627942f0 + 0.6357054f0im … -1.0425543f0 + 1.5618956f0im -0.38096747f0 + 0.8508501f0im; -0.45127892f0 + 0.8231804f0im 0.2312188f0 + 1.1413448f0im … -0.5119798f0 + 1.5518585f0im -0.1485673f0 + 1.0093914f0im]
Stacktrace:
 [1] record(ts::Test.DefaultTestSet, t::Union{Test.Error, Test.Fail})
   @ Test /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:983
 [2] top-level scope
   @ ~/.julia/packages/CUDA/ifpPv/test/runtests.jl:524
 [3] include(fname::String)
   @ Base.MainInclude ./client.jl:476
 [4] top-level scope
   @ none:6
 [5] eval
   @ ./boot.jl:368 [inlined]
 [6] exec_options(opts::Base.JLOptions)
   @ Base ./client.jl:276
 [7] _start()
   @ Base ./client.jl:522
cublas: Test Failed at /home/sunjin/.julia/packages/CUDA/ifpPv/test/cublas.jl:1031
  Expression: C ≈ Array(dA)
   Evaluated: ...
Stacktrace:
 [1] record(ts::Test.DefaultTestSet, t::Union{Test.Error, Test.Fail})
   @ Test /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:983
 [2] top-level scope
   @ ~/.julia/packages/CUDA/ifpPv/test/runtests.jl:524
 [3] include(fname::String)
   @ Base.MainInclude ./client.jl:476
 [4] top-level scope
   @ none:6
 [5] eval
   @ ./boot.jl:368 [inlined]
 [6] exec_options(opts::Base.JLOptions)
   @ Base ./client.jl:276
 [7] _start()
   @ Base ./client.jl:522
Worker 10 failed running test execution:
Some tests did not pass: 65 passed, 0 failed, 1 errored, 0 broken.
execution: Error During Test at /home/sunjin/.julia/packages/CUDA/ifpPv/test/execution.jl:55
  Got exception outside of a @test
  CUPTIError: CUPTI is unable to initialize its connection to the CUDA driver (code 15, CUPTI_ERROR_NOT_INITIALIZED)
  Stacktrace:
    [1] (::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(dummy), Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}})()
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:76
    [2] lock(f::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(dummy), Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}}, l::ReentrantLock)
      @ Base ./lock.jl:185
    [3] #code_sass#228
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:69 [inlined]
    [4] code_sass(io::Base.DevNull, func::Any, types::Any, kernel::Bool; verbose::Bool, always_inline::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:51
    [5] code_sass (repeats 2 times)
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:45 [inlined]
    [6] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/execution.jl:61 [inlined]
    [7] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
    [8] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/execution.jl:56 [inlined]
    [9] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [10] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/execution.jl:7
   [11] include
      @ ./client.jl:476 [inlined]
   [12] #12
      @ ~/.julia/packages/CUDA/ifpPv/test/runtests.jl:101 [inlined]
   [13] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [14] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [15] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [16] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/utilities.jl:25 [inlined]
   [17] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/pool.jl:573 [inlined]
   [18] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:54
   [19] eval
      @ ./boot.jl:368 [inlined]
   [20] runtests(f::Function, name::String, time_source::Symbol, snoop::Nothing)
      @ Main ~/.julia/packages/CUDA/ifpPv/test/setup.jl:66
   [21] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./essentials.jl:729
   [22] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:726
   [23] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285
   [24] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
   [25] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285 [inlined]
   [26] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:484
cusolver/sparse: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(7)
cusparse/linalg: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(12)
```

---

<div class="post-metadata">

**Author:** ![jinfreedom](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@jinfreedom](https://discourse.julialang.org/u/jinfreedom)\
**Post date:** [January 4, 2023, 2:49pm UTC](https://discourse.julialang.org/t/cuda-test-random-failure/92399/4 "2023-01-04T14:49:00Z")

</div>

Second half of the output

```julia

Test Summary: | Pass Fail Error Broken Total Time
  Overall | 15192 2 11 5 15210      
    initialization | 30 30      
    gpuarrays/indexing scalar | 476 476      
    gpuarrays/math/power | None      
    gpuarrays/linalg/mul!/vector-matrix | 168 168      
    gpuarrays/interface | 7 7      
    gpuarrays/indexing multidimensional | 46 46      
    gpuarrays/linalg | 231 231      
    gpuarrays/reductions/reducedim! | 192 192      
    gpuarrays/uniformscaling | 56 56      
    gpuarrays/reductions/any all count | 101 101      
    gpuarrays/math/intrinsics | 12 12      
    gpuarrays/statistics | 1 1      
    gpuarrays/linalg/mul!/matrix-matrix | 432 432      
    gpuarrays/constructors | 899 899      
    gpuarrays/random | 62 62      
    gpuarrays/linalg/norm | 1 1      
    gpuarrays/base | 75 75      
    gpuarrays/reductions/minimum maximum extrema | 666 666      
    gpuarrays/reductions/mapreduce | 396 396      
    gpuarrays/reductions/== isequal | 312 312      
    gpuarrays/reductions/reduce | 264 264      
    apiutils | 6 6      
    gpuarrays/reductions/mapreducedim! | 1 1      
    broadcast | 22 22      
    codegen | 13 3 16      
    gpuarrays/broadcasting | 1 1      
    array | 364 364      
    cudadrv | 139 1 140      
    curand | 1 1      
    cufft | 177 177      
    examples | 1 1      
    cusparse | 949 949      
    cublas | 2180 2 2182      
    execution | 65 1 66      
    gpuarrays/reductions/sum prod | 862 862      
    iterator | 37 37      
    nvtx | None      
    pointer | 35 35      
    exceptions | 17 17      
    pool | 10 10      
    nvml | 11 11      
    linalg | 21 21      
    random | 117 117      
    utils | 55 55      
    threading | None      
    cusolver/multigpu | 30 30      
    texture | 38 4 42      
    cusolver/sparse | 1 1      
    cusparse/conversions | 62 62      
    cusparse/device | 10 10      
    cusparse/broadcast | 65 65      
    sorting | 272 272      
    cusparse/generic | 1238 1238      
    device/array | 20 20      
    cusparse/interfaces | 1042 1042      
    device/intrinsics | 38 38      
    device/ldg | 22 22      
    cusparse/linalg | 1 1      
    cusolver/dense | 1940 1940      
    device/intrinsics/memory | 16 16      
┌ Info: System information:
│ CUDA runtime 11.8, artifact installation
│ CUDA driver 12.0
│ NVIDIA driver 527.56.0
│ 
│ Libraries: 
│ - CUBLAS: 11.11.3
│ - CURAND: 10.3.0
│ - CUFFT: 10.9.0
│ - CUSOLVER: 11.4.1
│ - CUSPARSE: 11.7.5
│ - CUPTI: 18.0.0
│ - NVML: 12.0.0+525.65
│ 
│ Toolchain:
│ - Julia: 1.8.1
│ - LLVM: 13.0.1
│ - PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0, 7.1, 7.2
│ - Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86
│ 
│ 1 device:
└ 0: NVIDIA GeForce RTX 3060 Laptop GPU (sm_86, 5.679 GiB / 6.000 GiB available)
[ Info: Testing using 1 device(s): 0. NVIDIA GeForce RTX 3060 Laptop GPU (UUID f780bc3c-a9f0-9c3d-e723-c01de3450fd7)
Worker 5 terminated.
UNHANDLED TASK ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
Worker 3 terminated.
UNHANDLED TASK ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
Worker 2 terminated.
UNHANDLED TASK ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
Worker 4 terminated.
UNHANDLED TASK ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
Worker 6 terminated.
UNHANDLED TASK ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
Worker 7 terminated.
UNHANDLED TASK ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
Worker 12 terminated.
UNHANDLED TASK ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:941
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:950
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
ERROR: LoadError: Test run finished with errors
in expression starting at /home/sunjin/.julia/packages/CUDA/ifpPv/test/runtests.jl:555
    device/random | 156 156      
    device/intrinsics/output | 40 40      
    device/intrinsics/atomics | 147 147      
    device/intrinsics/math | 104 104      
    device/intrinsics/wmma | 446 446      
    e[31;1mFAILUREe[0m

Error in testset gpuarrays/statistics:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(5)
Error in testset gpuarrays/linalg/norm:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(3)
Error in testset gpuarrays/reductions/mapreducedim!:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(2)
Error in testset codegen:
Error During Test at /home/sunjin/.julia/packages/CUDA/ifpPv/test/codegen.jl:170
  Test threw exception
  Expression: CUDA.code_sass(devnull, valid_kernel, Tuple{}) == nothing
  CUPTIError: CUPTI is unable to initialize its connection to the CUDA driver (code 15, CUPTI_ERROR_NOT_INITIALIZED)
  Stacktrace:
    [1] (::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{var"#valid_kernel#71", Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}})()
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:76
    [2] lock(f::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{var"#valid_kernel#71", Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}}, l::ReentrantLock)
      @ Base ./lock.jl:185
    [3] #code_sass#228
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:69 [inlined]
    [4] code_sass(io::Base.DevNull, func::Any, types::Any, kernel::Bool; verbose::Bool, always_inline::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:51
    [5] code_sass (repeats 2 times)
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:45 [inlined]
    [6] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:464 [inlined]
    [7] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:170 [inlined]
    [8] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
    [9] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:167 [inlined]
   [10] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [11] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:166
Error in testset codegen:
Error During Test at /home/sunjin/.julia/packages/CUDA/ifpPv/test/codegen.jl:174
  Got exception outside of a @test
  CUPTIError: CUPTI is unable to initialize its connection to the CUDA driver (code 15, CUPTI_ERROR_NOT_INITIALIZED)
  Stacktrace:
    [1] (::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(kernel_341), Tuple{Ptr{Int64}}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}})()
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:76
    [2] lock(f::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(kernel_341), Tuple{Ptr{Int64}}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}}, l::ReentrantLock)
      @ Base ./lock.jl:185
    [3] #code_sass#228
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:69 [inlined]
    [4] code_sass(io::Base.DevNull, func::Any, types::Any, kernel::Bool; verbose::Bool, always_inline::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:51
    [5] code_sass (repeats 2 times)
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:45 [inlined]
    [6] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:179 [inlined]
    [7] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
    [8] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:175 [inlined]
    [9] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [10] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:166
   [11] include
      @ ./client.jl:476 [inlined]
   [12] #12
      @ ~/.julia/packages/CUDA/ifpPv/test/runtests.jl:101 [inlined]
   [13] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [14] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [15] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [16] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/utilities.jl:25 [inlined]
   [17] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/pool.jl:573 [inlined]
   [18] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:54
   [19] eval
      @ ./boot.jl:368 [inlined]
   [20] runtests(f::Function, name::String, time_source::Symbol, snoop::Nothing)
      @ Main ~/.julia/packages/CUDA/ifpPv/test/setup.jl:66
   [21] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./essentials.jl:729
   [22] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:726
   [23] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285
   [24] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
   [25] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285 [inlined]
   [26] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:484
Error in testset codegen:
Error During Test at /home/sunjin/.julia/packages/CUDA/ifpPv/test/codegen.jl:182
  Got exception outside of a @test
  CUPTIError: CUPTI doesn't allow multiple callback subscribers. Only a single subscriber can be registered at a time. (code 39, CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED)
  Stacktrace:
    [1] (::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{var"#kernel#73", Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}})()
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:76
    [2] lock(f::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{var"#kernel#73", Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}}, l::ReentrantLock)
      @ Base ./lock.jl:185
    [3] #code_sass#228
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:69 [inlined]
    [4] code_sass(io::Base.DevNull, func::Any, types::Any, kernel::Bool; verbose::Bool, always_inline::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:51
    [5] code_sass (repeats 2 times)
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:45 [inlined]
    [6] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:185 [inlined]
    [7] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
    [8] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:183 [inlined]
    [9] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [10] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/codegen.jl:166
   [11] include
      @ ./client.jl:476 [inlined]
   [12] #12
      @ ~/.julia/packages/CUDA/ifpPv/test/runtests.jl:101 [inlined]
   [13] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [14] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [15] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [16] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/utilities.jl:25 [inlined]
   [17] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/pool.jl:573 [inlined]
   [18] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:54
   [19] eval
      @ ./boot.jl:368 [inlined]
   [20] runtests(f::Function, name::String, time_source::Symbol, snoop::Nothing)
      @ Main ~/.julia/packages/CUDA/ifpPv/test/setup.jl:66
   [21] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./essentials.jl:729
   [22] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:726
   [23] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285
   [24] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
   [25] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285 [inlined]
   [26] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:484
Error in testset gpuarrays/broadcasting:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(4)
Error in testset examples:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(6)
Error in testset cublas:
Test Failed at /home/sunjin/.julia/packages/CUDA/ifpPv/test/cublas.jl:1031
  Expression: C ≈ Array(dA)
   Evaluated: ComplexF32[...]
Error in testset execution:
Error During Test at /home/sunjin/.julia/packages/CUDA/ifpPv/test/execution.jl:55
  Got exception outside of a @test
  CUPTIError: CUPTI is unable to initialize its connection to the CUDA driver (code 15, CUPTI_ERROR_NOT_INITIALIZED)
  Stacktrace:
    [1] (::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(dummy), Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}})()
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:76
    [2] lock(f::CUDA.var"#229#231"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(dummy), Tuple{}}}, Ptr{Nothing}, Base.RefValue{Any}, NamedTuple{(:image, :entry, :external_gvars), Tuple{Vector{UInt8}, String, Vector{String}}}}, l::ReentrantLock)
      @ Base ./lock.jl:185
    [3] #code_sass#228
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:69 [inlined]
    [4] code_sass(io::Base.DevNull, func::Any, types::Any, kernel::Bool; verbose::Bool, always_inline::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ CUDA ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:51
    [5] code_sass (repeats 2 times)
      @ ~/.julia/packages/CUDA/ifpPv/src/compiler/reflection.jl:45 [inlined]
    [6] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/execution.jl:61 [inlined]
    [7] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
    [8] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/execution.jl:56 [inlined]
    [9] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [10] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/execution.jl:7
   [11] include
      @ ./client.jl:476 [inlined]
   [12] #12
      @ ~/.julia/packages/CUDA/ifpPv/test/runtests.jl:101 [inlined]
   [13] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [14] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [15] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:55 [inlined]
   [16] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/utilities.jl:25 [inlined]
   [17] macro expansion
      @ ~/.julia/packages/CUDA/ifpPv/src/pool.jl:573 [inlined]
   [18] top-level scope
      @ ~/.julia/packages/CUDA/ifpPv/test/setup.jl:54
   [19] eval
      @ ./boot.jl:368 [inlined]
   [20] runtests(f::Function, name::String, time_source::Symbol, snoop::Nothing)
      @ Main ~/.julia/packages/CUDA/ifpPv/test/setup.jl:66
   [21] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./essentials.jl:729
   [22] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:726
   [23] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285
   [24] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
   [25] macro expansion
      @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285 [inlined]
   [26] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:484
Error in testset cusolver/sparse:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(7)
Error in testset cusparse/linalg:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(12)
ERROR: LoadError: Package CUDA errored during testing
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
  [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
    @ Pkg.Operations /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1813
  [3] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::Base.Pairs{Symbol, IOStream, Tuple{Symbol}, NamedTuple{(:io,), Tuple{IOStream}}})
    @ Pkg.API /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/API.jl:431
  [4] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOStream, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/API.jl:156
  [5] test(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/API.jl:145
  [6] #test#87
    @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/API.jl:144 [inlined]
  [7] test
    @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/API.jl:144 [inlined]
  [8] #test#86
    @ /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/API.jl:143 [inlined]
  [9] test(pkg::String)
    @ Pkg.API /opt/julia-1.8.1/share/julia/stdlib/v1.8/Pkg/src/API.jl:143
 [10] top-level scope
    @ ~/julia/test.jl:4
in expression starting at /home/sunjin/julia/test.jl:4

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 4, 2023, 2:50pm UTC](https://discourse.julialang.org/t/cuda-test-random-failure/92399/5 "2023-01-04T14:50:01Z")

</div>

> [@jinfreedom](#):
>
> ` CUPTIError: CUPTI is unable to initialize its connection to the CUDA driver (code 15, CUPTI_ERROR_NOT_INITIALIZED)`

Something seems up with your driver, resulting in CUPTI not being available. This library is used for reflecting on native code, so its unavailability will break a bunch of tests.