PATH variable ignored when finding nsys executable (Windows 11)

Internals & Design
,
1

Running Julia 1.7.2 under Nsight Systems 2022.2.1 on Windows 11. When the CUDA.@profile macro is called, the nsys executable is not found, even though it’s on the PATH. Instead, the JULIA_CUDA_NSYS variable has to be explicitly set for it to work.

Details on reproducing and troubleshooting this:

  • nsys launch C:\Users\<username>\AppData\Local\Programs\Julia-1.7.2\bin\julia.exe
  • pkg> st

    [052768ef] CUDA v3.8.5

  • using CUDA
  • CuArray([1]).+1;
  • CUDA.@profile CuArray([1]).+1;

    Running under Nsight Systems, but could not find the nsys binary to start the profiler. Please specify using JULIA_CUDA_NSYS=path/to/nsys, and file an issue with the contents of ENV.

  • nsys_dir = "C:\\Program Files\\NVIDIA Corporation\\Nsight Systems 2022.2.1\\target-windows-x64";
  • julia> occursin(nsys_dir, ENV["PATH"])

    true

  • "JULIA_CUDA_NSYS" in keys(ENV)
    • false
  • ENV["JULIA_CUDA_NSYS"] = nsys_dir * "\\nsys.exe"

    "C:\\Program Files\\NVIDIA Corporation\\Nsight Systems 2022.2.1\\target-windows-x64\\nsys.exe"

  • julia> CUDA.@profile CuArray([1]).+1;

    ERROR: IOError: could not spawn ‘C:\Program Files\NVIDIA Corporation\Nsight Systems 2022.2.1\target-windows-x64’ start --capture-range=cudaProfilerApi: no such file or directory (ENOENT)

Note in this error that the JULIA_CUDA_NSYS variable appears to be set correctly but the executable name (but not directory path) seems to be stripped out when the CUDA.@profile macro is called.

However, if we set the variable to that same value in ~/.julia/config/startup.jl and restart Julia, then it works.

  • In ~/.julia/config/startup.jl
    • ENV["JULIA_CUDA_NSYS"] = "C:\\Program Files\\NVIDIA Corporation\\Nsight Systems 2022.2.1\\target-windows-x64\\nsys.exe"
  • nsys launch C:\Users\<username>\AppData\Local\Programs\Julia-1.7.2\bin\julia.exe
  • using CUDA
  • julia> CuArray([1]).+1;
  • julia> CUDA.@profile CuArray([1]).+1;

Works, profile is generated

Some contents of ENV

  • julia> filter(key -> startswith(key, "NSYS"), keys(ENV)) .|> key -> "$key => $(ENV[key])"

    6-element Vector{String}:
    “NSYS_CONTROL_CUPTI_FEATURES => 1”
    “NSYS_AGENT_TMP_DIR => C:\Users\\AppData\Local\Temp”
    “NSYS_CUDA_GPU_TIME_CONVERSION => raw”
    “NSYS_TARGET_STD_REDIRECT_ACTION => 3”
    “NSYS_PROFILING_SESSION_ID => 107676”
    “NSYS_TSC_SUPPORT => enable”

  • julia> filter(key -> occursin("CUDA", key), keys(ENV)) .|> key -> "$key => $(ENV[key])"

    7-element Vector{String}:
    “CUDA_INJECTION64_PATH => C:/Pro” ⋯ 62 bytes ⋯ “indows-x64\ToolsInjection64.dll”
    “JULIA_CUDA_NSYS => C:\Program F” ⋯ 44 bytes ⋯ “2.1\target-windows-x64\nsys.exe”
    “QUADD_CUDA_CONFIG => C:\Users\a” ⋯ 58 bytes ⋯ “files/injection_config_b33f575b”
    “JULIA_CUDA_USE_BINARYBUILDER => false”
    “CUDA_PATH => C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6”
    “NSYS_CUDA_GPU_TIME_CONVERSION => raw”
    “CUDA_PATH_V11_6 => C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6”

2

An actual issue on the CUDA.jl repo would have been better, but OK. Can you try https://github.com/JuliaGPU/CUDA.jl/pull/1459?

3

Awesome- I just tested it out, and it looks good except for one thing-- when splitting the environment variable contents on Windows, you’ll want to use semicolon not colon as the delimiter. I put a comment in the PR for that, but otherwise it works great! Nice idea using the path of the injected DLL to find the executable, to avoid version mismatches. Thanks again for looking at this and being so responsive!