CUDAnative test fails

eaubanel · April 23, 2020, 1:36pm

I’m experienced with C/CUDA, but trying CUDA with Julia for the first time, on a compute node of a research cluster running CentOS Linux release 7.6.1810 and eight Tesla V100-SXM2 GPUs:

nvidia-smi

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

I installed the CUDAdrv, CUDAnative, CuArrays packages successfully, and ran the Pkg.test on CUDAdrv and CuArrays successfully. However, the CUDAnative test wasn’t completely successfull:

julia> Pkg.test("CUDAnative")
    Testing CUDAnative
Status `/tmp/jl_0jNZ6d/Manifest.toml`
  [79e6a3ab] Adapt v1.0.1
  [b99e7846] BinaryProvider v0.5.8
  [fa961155] CEnum v0.2.0
  [3895d2a7] CUDAapi v4.0.0
  [c5f51814] CUDAdrv v6.2.2
  [be33ccc6] CUDAnative v3.0.4
  [da1fd8a2] CodeTracking v0.5.9
  [f68482b8] Cthulhu v1.0.2
  [864edb3b] DataStructures v0.17.13
  [e2ba6199] ExprTools v0.1.1
  [929cbde3] LLVM v1.3.4
  [bac558e1] OrderedCollections v1.1.0
  [a759f4b9] TimerOutputs v0.5.4
  [2a0f44e3] Base64 
  [ade2ca70] Dates 
  [8ba89e20] Distributed 
  [b77e0a4c] InteractiveUtils 
  [76f85450] LibGit2 
  [8f399da3] Libdl 
  [37e2e46d] LinearAlgebra 
  [56ddb016] Logging 
  [d6f4376e] Markdown 
  [44cfe95a] Pkg 
  [de0858da] Printf 
  [3fa0cd96] REPL 
  [9a3f8284] Random 
  [ea8e919c] SHA 
  [9e88b42a] Serialization 
  [6462fe0b] Sockets 
  [8dfed614] Test 
  [cf7118a7] UUIDs 
  [4ec0a83e] Unicode 
Downloading artifact: CUDA10.2

Downloading artifact: CUDA10.1

Downloading artifact: CUDA10.0

Downloading artifact: CUDA9.2

Downloading artifact: CUDA9.0

[ Info: Testing using device Tesla V100-SXM2-32GB (compute capability 7.0.0, 31.439 GiB available memory) on CUDA driver 10.2.0 and toolkit 10.0.130
stack traces at different debug levels: Test Failed at /home/eaubanel/.julia/packages/CUDAnative/ierw8/test/device/execution.jl:576
  Expression: isempty(out)
   Evaluated: isempty("Downloading artifact: CUDA10.2\n\e[?25l\e[1A\e[2K\e[?25hDownloading artifact: CUDA10.1\n\e[?25l\e[1A\e[2K\e[?25hDownloading artifact: CUDA10.0\n\e[?25l\e[1A\e[2K\e[?25hDownloading artifact: CUDA9.2\n\e[?25l\e[1A\e[2K\e[?25hDownloading artifact: CUDA9.0\n\e[?25l\e[1A\e[2K\e[?25h")
Stacktrace:
 [1] macro expansion at /home/eaubanel/.julia/packages/CUDAnative/ierw8/test/device/execution.jl:576 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113 [inlined]
 [3] macro expansion at /home/eaubanel/.julia/packages/CUDAnative/ierw8/test/device/execution.jl:560 [inlined]
 [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113 [inlined]
 [5] macro expansion at /home/eaubanel/.julia/packages/CUDAnative/ierw8/test/device/execution.jl:558 [inlined]
 [6] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113 [inlined]
 [7] top-level scope at /home/eaubanel/.julia/packages/CUDAnative/ierw8/test/device/execution.jl:5
Test Summary:                                | Pass  Fail  Total
CUDAnative                                   |  752     1    753
  base interface                             |             No tests
  pointer                                    |   20           20
  code generation                            |   84           84
  code generation (relying on a device)      |    8            8
  execution                                  |   89     1     90
    @cuda                                    |   14           14
    argument passing                         |   30           30
    exceptions                               |   16     1     17
      stack traces at different debug levels |   11     1     12
      #329                                   |    5            5
    shmem divergence bug                     |    7            7
    dynamic parallelism                      |   11           11
    cooperative groups                       |    1            1
    threading                                |   10           10
  pointer                                    |   37           37
  device arrays                              |   20           20
  CUDA functionality                         |  264          264
  WMMA                                       |  206          206
  NVTX                                       |             No tests
  examples                                   |    6            6
ERROR: LoadError: Some tests did not pass: 752 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /home/eaubanel/.julia/packages/CUDAnative/ierw8/test/runtests.jl:12
ERROR: Package CUDAnative errored during testing
Stacktrace:
 [1] pkgerror(::String, ::Vararg{String,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:53
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:1503
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:316
 [4] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:303
 [5] #test#68 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:297 [inlined]
 [6] test at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:297 [inlined]
 [7] #test#67 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:296 [inlined]
 [8] test at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:296 [inlined]
 [9] test(::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:295
 [10] test(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:295
 [11] top-level scope at REPL[4]:1

versioninfo()

Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, skylake)

I can compile and run C/CUDA programs just fine on this node. Can anyone help?

maleadt · April 23, 2020, 1:43pm

The package works, it just started downloading when doing an output check and stumbled on the message.

That said, it should never have attempted to download all artifacts, since they are marked lazy. Unless each of those downloads failed, could it be that you can’t download things on those compute nodes? If so it’s better to run with JULIA_CUDA_USE_BINARYBUILDER=false, or make sure you download the artifacts on the login node.

eaubanel · April 23, 2020, 1:59pm

Thanks, that was the problem. I used JULIA_CUDA_USE_BINARYBUILDER=false and the test passed.