Cannot build CUDAnative on login node (works on compute node)

samo · August 22, 2019, 4:58pm

Context

I am preparing a site-wide installation of Julia with the CUDA packages (CUDAdrv, CUDAnative and CuArrays) on our GPU supercomputer. To this purpose, I am creating automated installation recipes as for all our software stack. These recipes execute in an automatic fashion on the login nodes, which do not have any GPUs.

Problem

CUDAnative does not build on the login nodes. The reason seems to be that the installation procedure relies on being able to initialize CUDAdrv at the beginning. The error message is the following:

(v1.0.4-gpu) pkg> add CUDAnative
  Updating registry at `./registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed TimerOutputs ─────── v0.5.0
 Installed Adapt ────────────── v1.0.0
 Installed CEnum ────────────── v0.2.0
 Installed Crayons ──────────── v4.0.0
 Installed OrderedCollections ─ v1.1.0
 Installed DataStructures ───── v0.17.0
 Installed CUDAnative ───────── v2.2.1
 Installed LLVM ─────────────── v1.2.0
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [be33ccc6] + CUDAnative v2.2.1
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Manifest.toml`
  [79e6a3ab] + Adapt v1.0.0
  [fa961155] + CEnum v0.2.0
  [be33ccc6] + CUDAnative v2.2.1
  [a8cc5b0e] + Crayons v4.0.0
  [864edb3b] + DataStructures v0.17.0
  [929cbde3] + LLVM v1.2.0
  [bac558e1] + OrderedCollections v1.1.0
  [a759f4b9] + TimerOutputs v0.5.0
  [2a0f44e3] + Base64 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [37e2e46d] + LinearAlgebra 
  [d6f4376e] + Markdown 
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  Building CUDAnative → `/scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.log`
┌ Error: Error building `CUDAnative`: 
│ ERROR: LoadError: InitError: CUDA error: unknown error (code #999, ERROR_UNKNOWN)
│ Stacktrace:
│  [1] macro expansion at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/base.jl:145 [inlined]
│  [2] init(::Int64) at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/init.jl:10 (repeats 2 times)
│  [3] __init__() at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/CUDAdrv.jl:83
│  [4] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:633
│  [5] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:713
│  [6] _require(::Base.PkgId) at ./loading.jl:937
│  [7] require(::Base.PkgId) at ./loading.jl:858
│  [8] require(::Module, ::Symbol) at ./loading.jl:853
│  [9] include at ./boot.jl:317 [inlined]
│  [10] include_relative(::Module, ::String) at ./loading.jl:1044
│  [11] include(::Module, ::String) at ./sysimg.jl:29
│  [12] include(::String) at ./client.jl:392
│  [13] top-level scope at none:0
│ during initialization of module CUDAdrv
│ in expression starting at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.jl:2
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1096

Note that CUDAdrv itself builds properly on the login nodes. CUDAnative builds only correctly on the compute nodes. For details, see the shell output in the appendix at the end of this post.

Question

Can we avoid the initialization of CUDAdrv when building CUDAnative in order to enable building it on the login nodes? – Having to build CUDAnative on the compute nodes would be a severe complification of the automated site-wide installation with some additional drawbacks.

Appendix

Shell output of 1) successful installation of CUDAdrv on login node, 2) successful precompilation of CUDAdrv on compute node, 3) failing installation of CUDAnative on login node, 4) failing using CUDAdrv on login node, 5) successful build of CUDAnative on compute node, 6) successful precompilation of CUDAnative on compute node:

omlins@daint101:~> module use /scratch/snx3000/omlins/9_soft_install_julia/modules/all
omlins@daint101:~> module load julia/1.0.4 
omlins@daint101:~> cd /scratch/snx3000/omlins/julia_cuda_test_install
omlins@daint101:/scratch/snx3000/omlins/julia_cuda_test_install> export JULIA_DEPOT_PATH=.
omlins@daint101:/scratch/snx3000/omlins/julia_cuda_test_install> julia
(v1.0.4-gpu) pkg> status
    Status `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  (empty environment)

(v1.0.4-gpu) pkg> add CUDAdrv
   Cloning default registries into /scratch/snx3000/omlins/julia_cuda_test_install/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
 Resolving package versions...
 Installed CUDAapi ─ v1.0.1
 Installed CUDAdrv ─ v3.1.0
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [c5f51814] + CUDAdrv v3.1.0
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Manifest.toml`
  [3895d2a7] + CUDAapi v1.0.1
  [c5f51814] + CUDAdrv v3.1.0
  [8f399da3] + Libdl 
  [56ddb016] + Logging 
  [de0858da] + Printf 
  [4ec0a83e] + Unicode 

(v1.0.4-gpu) pkg> 
omlins@daint101:/scratch/snx3000/omlins/julia_cuda_test_install> srun -Cgpu -pdebug --pty bash
srun: job 16276058 queued and waiting for resources
srun: job 16276058 has been allocated resources
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> module unload julia
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> module load julia/1.0.4 
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> export JULIA_DEPOT_PATH=.
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> julia
(v1.0.4-gpu) pkg> status
    Status `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [c5f51814] CUDAdrv v3.1.0

julia> using CUDAdrv
[ Info: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde]

julia> 
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> exit
omlins@daint101:/scratch/snx3000/omlins/julia_cuda_test_install> julia
(v1.0.4-gpu) pkg> status
    Status `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [c5f51814] CUDAdrv v3.1.0

(v1.0.4-gpu) pkg> add CUDAnative
  Updating registry at `./registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed TimerOutputs ─────── v0.5.0
 Installed Adapt ────────────── v1.0.0
 Installed CEnum ────────────── v0.2.0
 Installed Crayons ──────────── v4.0.0
 Installed OrderedCollections ─ v1.1.0
 Installed DataStructures ───── v0.17.0
 Installed CUDAnative ───────── v2.2.1
 Installed LLVM ─────────────── v1.2.0
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [be33ccc6] + CUDAnative v2.2.1
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Manifest.toml`
  [79e6a3ab] + Adapt v1.0.0
  [fa961155] + CEnum v0.2.0
  [be33ccc6] + CUDAnative v2.2.1
  [a8cc5b0e] + Crayons v4.0.0
  [864edb3b] + DataStructures v0.17.0
  [929cbde3] + LLVM v1.2.0
  [bac558e1] + OrderedCollections v1.1.0
  [a759f4b9] + TimerOutputs v0.5.0
  [2a0f44e3] + Base64 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [37e2e46d] + LinearAlgebra 
  [d6f4376e] + Markdown 
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  Building CUDAnative → `/scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.log`
┌ Error: Error building `CUDAnative`: 
│ ERROR: LoadError: InitError: CUDA error: unknown error (code #999, ERROR_UNKNOWN)
│ Stacktrace:
│  [1] macro expansion at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/base.jl:145 [inlined]
│  [2] init(::Int64) at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/init.jl:10 (repeats 2 times)
│  [3] __init__() at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/CUDAdrv.jl:83
│  [4] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:633
│  [5] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:713
│  [6] _require(::Base.PkgId) at ./loading.jl:937
│  [7] require(::Base.PkgId) at ./loading.jl:858
│  [8] require(::Module, ::Symbol) at ./loading.jl:853
│  [9] include at ./boot.jl:317 [inlined]
│  [10] include_relative(::Module, ::String) at ./loading.jl:1044
│  [11] include(::Module, ::String) at ./sysimg.jl:29
│  [12] include(::String) at ./client.jl:392
│  [13] top-level scope at none:0
│ during initialization of module CUDAdrv
│ in expression starting at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.jl:2
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1096

julia> using CUDAdrv
ERROR: InitError: CUDA error: unknown error (code #999, ERROR_UNKNOWN)
Stacktrace:
 [1] macro expansion at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/base.jl:145 [inlined]
 [2] init(::Int64) at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/init.jl:10 (repeats 2 times)
 [3] __init__() at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/CUDAdrv.jl:83
 [4] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:633
 [5] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:713
 [6] _require(::Base.PkgId) at ./loading.jl:937
 [7] require(::Base.PkgId) at ./loading.jl:858
 [8] require(::Module, ::Symbol) at ./loading.jl:853
during initialization of module CUDAdrv

julia> 
omlins@daint101:/scratch/snx3000/omlins/julia_cuda_test_install> srun -Cgpu -pdebug --pty bash
srun: job 16276186 queued and waiting for resources
srun: job 16276186 has been allocated resources
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> module unload julia
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> module load julia/1.0.4 
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> export JULIA_DEPOT_PATH=.
omlins@nid03511:/scratch/snx3000/omlins/julia_cuda_test_install> julia
(v1.0.4-gpu) pkg> status
    Status `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [c5f51814] CUDAdrv v3.1.0
  [be33ccc6] CUDAnative v2.2.1

(v1.0.4-gpu) pkg> build CUDAnative
  Building CUDAnative → `/scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.log`

(v1.0.4-gpu) pkg> 

julia> using CUDAnative
[ Info: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17]

julia>

samo · August 23, 2019, 8:44am

I could get CUDAnative to build on the login node by modifying the file packages/CUDAnative/nItlk/deps/build.jl as follows:

remove using CUDAdrv
replace CUDAdrv.configured with true
replace CUDAdrv.version() with v"9.2.0"

@maleadt, could you modify the build procedure of CUDAnative to obtain this values without requiring to initialize CUDAdrv?

Shell output shared here for completeness:

omlins@daint105:/scratch/snx3000/omlins/julia_cuda_test_install> cp /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.jl .
omlins@daint105:/scratch/snx3000/omlins/julia_cuda_test_install> rm -rf compiled/ conda/ environments/ logs/ packages/ prefs/ registries/
omlins@daint105:/scratch/snx3000/omlins/julia_cuda_test_install> julia
(v1.0.4-gpu) pkg> status
    Status `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  (empty environment)

(v1.0.4-gpu) pkg> add CUDAdrv
   Cloning default registries into /scratch/snx3000/omlins/julia_cuda_test_install/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
 Resolving package versions...
 Installed CUDAapi ─ v1.0.1
 Installed CUDAdrv ─ v3.1.0
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [c5f51814] + CUDAdrv v3.1.0
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Manifest.toml`
  [3895d2a7] + CUDAapi v1.0.1
  [c5f51814] + CUDAdrv v3.1.0
  [8f399da3] + Libdl 
  [56ddb016] + Logging 
  [de0858da] + Printf 
  [4ec0a83e] + Unicode 

(v1.0.4-gpu) pkg> add CUDAnative
 Resolving package versions...
 Installed Adapt ────────────── v1.0.0
 Installed Crayons ──────────── v4.0.0
 Installed OrderedCollections ─ v1.1.0
 Installed CEnum ────────────── v0.2.0
 Installed TimerOutputs ─────── v0.5.0
 Installed DataStructures ───── v0.17.0
 Installed CUDAnative ───────── v2.2.1
 Installed LLVM ─────────────── v1.2.0
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [be33ccc6] + CUDAnative v2.2.1
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Manifest.toml`
  [79e6a3ab] + Adapt v1.0.0
  [fa961155] + CEnum v0.2.0
  [be33ccc6] + CUDAnative v2.2.1
  [a8cc5b0e] + Crayons v4.0.0
  [864edb3b] + DataStructures v0.17.0
  [929cbde3] + LLVM v1.2.0
  [bac558e1] + OrderedCollections v1.1.0
  [a759f4b9] + TimerOutputs v0.5.0
  [2a0f44e3] + Base64 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [37e2e46d] + LinearAlgebra 
  [d6f4376e] + Markdown 
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  Building CUDAnative → `/scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.log`
┌ Error: Error building `CUDAnative`: 
│ ERROR: LoadError: InitError: CUDA error: unknown error (code #999, ERROR_UNKNOWN)
│ Stacktrace:
│  [1] macro expansion at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/base.jl:145 [inlined]
│  [2] init(::Int64) at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/init.jl:10 (repeats 2 times)
│  [3] __init__() at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAdrv/ADRHQ/src/CUDAdrv.jl:83
│  [4] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:633
│  [5] _require_from_serialized(::String) at ./loading.jl:684
│  [6] _require(::Base.PkgId) at ./loading.jl:967
│  [7] require(::Base.PkgId) at ./loading.jl:858
│  [8] require(::Module, ::Symbol) at ./loading.jl:853
│  [9] include at ./boot.jl:317 [inlined]
│  [10] include_relative(::Module, ::String) at ./loading.jl:1044
│  [11] include(::Module, ::String) at ./sysimg.jl:29
│  [12] include(::String) at ./client.jl:392
│  [13] top-level scope at none:0
│ during initialization of module CUDAdrv
│ in expression starting at /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.jl:2
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1096

shell> cp -f build.jl /scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.jl

(v1.0.4-gpu) pkg> build CUDAnative
  Building CUDAnative → `/scratch/snx3000/omlins/julia_cuda_test_install/packages/CUDAnative/nItlk/deps/build.log`

(v1.0.4-gpu) pkg> 
omlins@daint105:/scratch/snx3000/omlins/julia_cuda_test_install> srun -Cgpu -pdebug --pty bash

srun: job 16300107 queued and waiting for resources
srun: job 16300107 has been allocated resources

omlins@nid03510:/scratch/snx3000/omlins/julia_cuda_test_install> 
omlins@nid03510:/scratch/snx3000/omlins/julia_cuda_test_install> module unload julia
omlins@nid03510:/scratch/snx3000/omlins/julia_cuda_test_install> module load julia/1.0.4 
omlins@nid03510:/scratch/snx3000/omlins/julia_cuda_test_install> export JULIA_DEPOT_PATH=.
omlins@nid03510:/scratch/snx3000/omlins/julia_cuda_test_install> julia
(v1.0.4-gpu) pkg> status
    Status `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [c5f51814] CUDAdrv v3.1.0
  [be33ccc6] CUDAnative v2.2.1

julia> using CUDAdrv

julia> using CUDAnative
[ Info: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17]

julia>

maleadt · August 23, 2019, 8:46am

I’ve recently done exactly that: Replace Pkg.build with runtime initialization by maleadt · Pull Request #435 · JuliaGPU/CUDAnative.jl · GitHub

samo · August 23, 2019, 10:34am

Thanks @malead! Installing the master version of the CUDA packages worked on the login nodes out of the box without any patching [1]
Unfortunately, there comes up another problem, which is: Precompiling CUDAnative on login node (works on compute node) .

[1] Shell output of successful installation of the CUDA packages on the login node:

omlins@daint104:~> module use /scratch/snx3000/omlins/9_soft_install_julia/modules/all
omlins@daint104:~> module load julia/1.0.4 
omlins@daint104:~> cd /scratch/snx3000/omlins/julia_cuda_test_install
omlins@daint104:/scratch/snx3000/omlins/julia_cuda_test_install> export JULIA_DEPOT_PATH=.
omlins@daint104:/scratch/snx3000/omlins/julia_cuda_test_install> julia
(v1.0.4-gpu) pkg> status
    Status `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  (empty environment)

(v1.0.4-gpu) pkg> add CUDAdrv#master
   Cloning default registries into /scratch/snx3000/omlins/julia_cuda_test_install/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
   Cloning git-repo `https://github.com/JuliaGPU/CUDAdrv.jl.git`
  Updating git-repo `https://github.com/JuliaGPU/CUDAdrv.jl.git`
 Resolving package versions...
 Installed CUDAapi ─ v1.0.1
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [c5f51814] + CUDAdrv v3.1.0 #master (https://github.com/JuliaGPU/CUDAdrv.jl.git)
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Manifest.toml`
  [3895d2a7] + CUDAapi v1.0.1
  [c5f51814] + CUDAdrv v3.1.0 #master (https://github.com/JuliaGPU/CUDAdrv.jl.git)
  [8f399da3] + Libdl 
  [56ddb016] + Logging 
  [de0858da] + Printf 
  [4ec0a83e] + Unicode 

(v1.0.4-gpu) pkg> add CUDAnative#master
   Cloning git-repo `https://github.com/JuliaGPU/CUDAnative.jl.git`
  Updating git-repo `https://github.com/JuliaGPU/CUDAnative.jl.git`
 Resolving package versions...
 Installed Adapt ────────────── v1.0.0
 Installed CEnum ────────────── v0.2.0
 Installed TimerOutputs ─────── v0.5.0
 Installed Crayons ──────────── v4.0.0
 Installed OrderedCollections ─ v1.1.0
 Installed DataStructures ───── v0.17.0
 Installed LLVM ─────────────── v1.2.0
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [be33ccc6] + CUDAnative v2.2.1 #master (https://github.com/JuliaGPU/CUDAnative.jl.git)
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Manifest.toml`
  [79e6a3ab] + Adapt v1.0.0
  [fa961155] + CEnum v0.2.0
  [be33ccc6] + CUDAnative v2.2.1 #master (https://github.com/JuliaGPU/CUDAnative.jl.git)
  [a8cc5b0e] + Crayons v4.0.0
  [864edb3b] + DataStructures v0.17.0
  [929cbde3] + LLVM v1.2.0
  [bac558e1] + OrderedCollections v1.1.0
  [a759f4b9] + TimerOutputs v0.5.0
  [2a0f44e3] + Base64 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [37e2e46d] + LinearAlgebra 
  [d6f4376e] + Markdown 
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 

(v1.0.4-gpu) pkg> add CuArrays#master
   Cloning git-repo `https://github.com/JuliaGPU/CuArrays.jl.git`
  Updating git-repo `https://github.com/JuliaGPU/CuArrays.jl.git`
 Resolving package versions...
 Installed Reexport ─────── v0.2.0
 Installed Requires ─────── v0.5.2
 Installed VersionParsing ─ v1.1.3
 Installed NNlib ────────── v0.6.0
 Installed Tokenize ─────── v0.5.6
 Installed FFTW ─────────── v0.3.0
 Installed GPUArrays ────── v1.0.0
 Installed StaticArrays ─── v0.11.0
 Installed BinaryProvider ─ v0.5.6
 Installed AbstractFFTs ─── v0.4.1
 Installed Compat ───────── v2.1.0
 Installed JSON ─────────── v0.21.0
 Installed FillArrays ───── v0.6.4
 Installed CSTParser ────── v0.6.2
 Installed Conda ────────── v1.3.0
 Installed Parsers ──────── v0.3.6
 Installed MacroTools ───── v0.5.1
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Project.toml`
  [3a865a2d] + CuArrays v1.1.0 #master (https://github.com/JuliaGPU/CuArrays.jl.git)
  Updating `/scratch/snx3000/omlins/julia_cuda_test_install/environments/v1.0.4-gpu/Manifest.toml`
  [621f4979] + AbstractFFTs v0.4.1
  [b99e7846] + BinaryProvider v0.5.6
  [00ebfdb7] + CSTParser v0.6.2
  [34da2185] + Compat v2.1.0
  [8f4d0f93] + Conda v1.3.0
  [3a865a2d] + CuArrays v1.1.0 #master (https://github.com/JuliaGPU/CuArrays.jl.git)
  [7a1cc6ca] + FFTW v0.3.0
  [1a297f60] + FillArrays v0.6.4
  [0c68f7d7] + GPUArrays v1.0.0
  [682c06a0] + JSON v0.21.0
  [1914dd2f] + MacroTools v0.5.1
  [872c559c] + NNlib v0.6.0
  [69de0a69] + Parsers v0.3.6
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v0.5.2
  [90137ffa] + StaticArrays v0.11.0
  [0796e94c] + Tokenize v0.5.6
  [81def892] + VersionParsing v1.1.3
  [ade2ca70] + Dates 
  [8bb1440f] + DelimitedFiles 
  [76f85450] + LibGit2 
  [a63ad114] + Mmap 
  [44cfe95a] + Pkg 
  [3fa0cd96] + REPL 
  [ea8e919c] + SHA 
  [1a1011a3] + SharedArrays 
  [2f01184e] + SparseArrays 
  [10745b16] + Statistics 
  [cf7118a7] + UUIDs 
  Building Conda → `/scratch/snx3000/omlins/julia_cuda_test_install/packages/Conda/kLXeC/deps/build.log`
  Building FFTW ─→ `/scratch/snx3000/omlins/julia_cuda_test_install/packages/FFTW/2okGQ/deps/build.log`