Flux failing on GPU

Specific Domains Machine Learning
21

From what I read in the notebook, seems like cudnn isn’t from version 7.6 or higher. Have you validated CUDNN version shipped on AWS instance? For example:
cat /usr/local/cuda-10.1/include/cudnn.h | grep CUDNN_MAJOR -A 2.
If it’s effectively version 7.5.1 as indicated in warning messages, then I guess an upgrade of CUDNN might make it.

1 Like
22

You are using Flux 0.8.3, which had some bugs wrt. CUDA usage. Please upgrade that package.

1 Like
23

ok, I can do that:


$ cat /usr/local/cuda-10.1/include/cudnn.h | grep CUDNN_MAJOR

#define **CUDNN_MAJOR** 7

#define CUDNN_VERSION ( **CUDNN_MAJOR** * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)
$
24

Hm. I had some difficulties doing that. Trace below. I removed all packages, then did Pkg.update(), then tried to add 1.10.1 which failed due to dependency issues. Then I tried the Pkg.add(“Flux”) command, which installed 0.8.3.

Do you have any hints on how I should proceed?

julia> Pkg.update()

Updating registry at ~/.julia/registries/General

Updating git-repo https://github.com/JuliaRegistries/General.git

Resolving package versions…

Installed ImageShow ─────────────────── v0.2.3

Installed Plotly ────────────────────── v0.3.0

Installed IJulia ────────────────────── v1.21.1

Installed BenchmarkTools ────────────── v0.4.3

Installed CUDAapi ───────────────────── v3.1.0

Installed CUDAdrv ───────────────────── v6.0.0

Installed CuArrays ──────────────────── v1.7.2

Installed BSON ──────────────────────── v0.2.5

Installed ImageMagick ───────────────── v1.1.2

Installed CUDAnative ────────────────── v2.10.2

Installed Plots ─────────────────────── v0.29.1

Installed WebIO ─────────────────────── v0.8.13

Installed GPUArrays ─────────────────── v2.0.1

Installed Colors ────────────────────── v0.11.2

Installed OrdinaryDiffEq ────────────── v5.29.0

Installed FileIO ────────────────────── v1.2.2

Installed Zstd_jll ──────────────────── v1.4.4+0

Installed DiffResults ───────────────── v1.0.2

Installed NaNMath ───────────────────── v0.3.3

Installed TimerOutputs ──────────────── v0.5.3

Installed DiffRules ─────────────────── v1.0.1

Installed DifferentialEquations ─────── v6.11.0

Installed PlotUtils ─────────────────── v0.6.3

Installed StatsBase ─────────────────── v0.32.1

Installed AssetRegistry ─────────────── v0.1.0

Installed Widgets ───────────────────── v0.6.2

Installed ZMQ ───────────────────────── v1.2.0

Installed Measures ──────────────────── v0.3.1

Installed VertexSafeGraphs ──────────── v0.1.1

Installed PaddedViews ───────────────── v0.5.1

Installed libpng_jll ────────────────── v1.6.37+2

Installed MbedTLS ───────────────────── v0.7.0

Installed GeometryTypes ─────────────── v0.7.7

Installed LightGraphs ───────────────── v1.3.0

Installed MacroTools ────────────────── v0.5.4

Installed NLSolversBase ─────────────── v7.6.1

Installed FFMPEG ────────────────────── v0.2.4

Installed ExponentialUtilities ──────── v1.6.0

Installed FunctionWrappers ──────────── v1.0.0

Installed ImageMagick_jll ───────────── v6.9.10-12+0

Installed NNlib ─────────────────────── v0.6.4

Installed ArrayInterface ────────────── v2.4.1

Installed FillArrays ────────────────── v0.8.4

Installed HTTP ──────────────────────── v0.8.8

Installed Roots ─────────────────────── v0.8.4

Installed ImageCore ─────────────────── v0.8.11

Installed BinDeps ───────────────────── v1.0.0

Installed DataAPI ───────────────────── v1.1.0

Installed PlotThemes ────────────────── v1.0.1

Installed DelayDiffEq ───────────────── v5.21.0

Installed Showoff ───────────────────── v0.3.1

Installed Pidfile ───────────────────── v1.1.0

Installed DataValueInterfaces ───────── v1.0.0

Installed StaticArrays ──────────────── v0.12.1

Installed DiffEqFinancial ───────────── v2.2.1

Installed SimpleTraits ──────────────── v0.9.1

Installed PlotlyBase ────────────────── v0.3.0

Installed MappedArrays ──────────────── v0.2.2

Installed FiniteDiff ────────────────── v2.2.0

Installed DiffEqJump ────────────────── v6.5.0

Installed Requires ──────────────────── v1.0.1

Installed RecipesBase ───────────────── v0.8.0

Installed ArrayLayouts ──────────────── v0.1.5

Installed RecursiveFactorization ────── v0.1.0

Installed DataStructures ────────────── v0.17.9

Installed RandomNumbers ─────────────── v1.4.0

Installed Distances ─────────────────── v0.8.2

Installed Zlib_jll ──────────────────── v1.2.11+8

Installed SpecialFunctions ──────────── v0.10.0

Installed ChainRulesCore ────────────── v0.6.1

Installed Graphics ──────────────────── v1.0.1

Installed ArnoldiMethod ─────────────── v0.0.4

Installed IteratorInterfaceExtensions ─ v1.0.0

Installed DiffEqNoiseProcess ────────── v3.9.0

Installed Hiccup ────────────────────── v0.2.2

Installed CEnum ─────────────────────── v0.2.0

Installed VersionParsing ────────────── v1.2.0

Installed PlotlyJS ──────────────────── v0.13.1

Installed ColorTypes ────────────────── v0.9.1

Installed SteadyStateDiffEq ─────────── v1.5.0

Installed GR ────────────────────────── v0.46.0

Installed ForwardDiff ───────────────── v0.10.9

Installed Compat ────────────────────── v2.2.0

Installed OffsetArrays ──────────────── v1.0.2

Installed JSExpr ────────────────────── v0.5.1

Installed RecursiveArrayTools ───────── v2.1.0

Installed GenericSVD ────────────────── v0.2.2

Installed FunctionalCollections ─────── v0.5.0

Installed FixedPointNumbers ─────────── v0.7.1

Installed WebSockets ────────────────── v1.5.2

Installed AbstractFFTs ──────────────── v0.5.0

Installed Parsers ───────────────────── v0.3.11

Installed ResettableStacks ──────────── v1.0.0

Installed ZygoteRules ───────────────── v0.2.0

Installed Sundials ──────────────────── v3.8.2

Installed DiffEqBase ────────────────── v6.16.0

Installed Inflate ───────────────────── v0.1.1

Installed JSON ──────────────────────── v0.21.0

Installed SortingAlgorithms ─────────── v0.3.1

Installed Mustache ──────────────────── v0.5.13

Installed LLVM ──────────────────────── v1.3.3

Installed TableTraits ───────────────── v1.0.0

Installed Tables ────────────────────── v1.0.1

Installed DiffEqPhysics ─────────────── v3.5.0

Installed ZeroMQ_jll ────────────────── v4.3.2+0

Installed Adapt ─────────────────────── v1.0.1

Installed BandedMatrices ────────────── v0.14.3

Installed DiffEqCallbacks ───────────── v2.12.0

Installed Reexport ──────────────────── v0.2.0

Installed DimensionalPlotRecipes ────── v1.1.0

Installed LineSearches ──────────────── v7.0.1

Installed Conda ─────────────────────── v1.4.1

Installed SparseDiffTools ───────────── v1.3.3

Installed URIParser ─────────────────── v0.4.0

Installed BinaryProvider ────────────── v0.5.8

Installed OrderedCollections ────────── v1.1.0

Installed LaTeXStrings ──────────────── v1.0.3

Installed IniFile ───────────────────── v0.5.0

Installed SoftGlobalScope ───────────── v1.0.10

Installed Parameters ────────────────── v0.12.0

Installed NLsolve ───────────────────── v4.3.0

Installed MuladdMacro ───────────────── v0.2.2

Installed BoundaryValueDiffEq ───────── v2.4.0

Installed OpenSpecFun_jll ───────────── v0.5.3+1

Installed DocStringExtensions ───────── v0.8.1

Installed Blink ─────────────────────── v0.12.1

Installed CommonSubexpressions ──────── v0.2.0

Installed StochasticDiffEq ──────────── v6.19.1

Installed IterativeSolvers ──────────── v0.8.1

Installed JpegTurbo_jll ─────────────── v2.0.1+0

Installed Libtiff_jll ───────────────── v4.0.10+0

Installed Missings ──────────────────── v0.4.3

Installed Contour ───────────────────── v0.5.1

Installed TreeViews ─────────────────── v0.3.0

Installed Observables ───────────────── v0.2.3

Installed Mux ───────────────────────── v0.7.1

Installed Lazy ──────────────────────── v0.14.0

Installed PoissonRandom ─────────────── v0.4.0

Installed MultiScaleArrays ──────────── v1.6.0

Updating ~/.julia/environments/v1.3/Project.toml

[no changes]

Updating ~/.julia/environments/v1.3/Manifest.toml

[no changes]

Building Conda ───→ ~/.julia/packages/Conda/3rPhK/deps/build.log

Building MbedTLS ─→ ~/.julia/packages/MbedTLS/a1JFn/deps/build.log

Building IJulia ──→ ~/.julia/packages/IJulia/yLI42/deps/build.log

Building GR ──────→ ~/.julia/packages/GR/NSt7D/deps/build.log

Building FFMPEG ──→ ~/.julia/packages/FFMPEG/guN1x/deps/build.log

Building Plots ───→ ~/.julia/packages/Plots/12uaJ/deps/build.log

Building WebIO ───→ ~/.julia/packages/WebIO/2nnB1/deps/build.log

Building NNlib ───→ ~/.julia/packages/NNlib/3krvM/deps/build.log

Building Blink ───→ ~/.julia/packages/Blink/4FM8Y/deps/build.log

Building PlotlyJS → ~/.julia/packages/PlotlyJS/AhkM5/deps/build.log

Building Sundials → ~/.julia/packages/Sundials/GuMs0/deps/build.log

julia> Pkg.add(Pkg.PackageSpec(;name=“Flux”, version=“0.10.1”))

Resolving package versions…

ERROR: Unsatisfiable requirements detected for package Flux [587475ba]:

Flux [587475ba] log:

├─possible versions are: [0.4.1, 0.5.0-0.5.4, 0.6.0-0.6.10, 0.7.0-0.7.3, 0.8.0-0.8.3, 0.9.0, 0.10.0-0.10.1] or uninstalled

├─restricted to versions 0.10.1 by an explicit requirement, leaving only versions 0.10.1

└─restricted by compatibility requirements with Colors [5ae59095] to versions: [0.4.1, 0.5.0-0.5.4, 0.6.0-0.6.10, 0.7.0-0.7.3, 0.8.0-0.8.3, 0.9.0] or uninstalled — no versions left

└─Colors [5ae59095] log:

├─possible versions are: [0.9.0, 0.9.2-0.9.6, 0.10.0-0.10.2, 0.11.0-0.11.2] or uninstalled

└─restricted to versions 0.11.2 by an explicit requirement, leaving only versions 0.11.2

Stacktrace:

[1] #propagate_constraints!#61( ::Bool, ::typeof(Pkg.GraphType.propagate_constraints!), ::Pkg.GraphType.Graph, ::Set{Int64} ) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/GraphType.jl:1007

[2] propagate_constraints! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/GraphType.jl:948 [inlined]

[3] #simplify_graph!#121( ::Bool, ::typeof(Pkg.GraphType.simplify_graph!), ::Pkg.GraphType.Graph, ::Set{Int64} ) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/GraphType.jl:1462

[4] simplify_graph! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/GraphType.jl:1462 [inlined] (repeats 2 times)

[5] resolve_versions!( ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1} ) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Operations.jl:321

[6] #add#112( ::Bool, ::Pkg.BinaryPlatforms.Linux, ::typeof(Pkg.Operations.add), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Array{Base.UUID,1} ) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Operations.jl:1010

[7] #add at ./none:0 [inlined]

[8] #add#25( ::Bool, ::Pkg.BinaryPlatforms.Linux, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Pkg.API.add), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1} ) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:102

[9] add( ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1} ) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:72

[10] #add#24 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:69 [inlined]

[11] add at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:69 [inlined]

[12] #add#20 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:66 [inlined]

[13] add( ::Pkg.Types.PackageSpec ) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:66

[14] top-level scope at REPL[8]:1

julia> Pkg.add(“Flux”)

Resolving package versions…

Installed Juno ─────────────── v0.8.0

Installed AbstractTrees ────── v0.3.2

Installed CodecZlib ────────── v0.6.0

Installed ZipFile ──────────── v0.9.1

Installed Tracker ──────────── v0.2.6

Installed Media ────────────── v0.5.0

Installed Flux ─────────────── v0.8.3

Installed TranscodingStreams ─ v0.9.5

Updating ~/.julia/environments/v1.3/Project.toml

[587475ba] + Flux v0.8.3

Updating ~/.julia/environments/v1.3/Manifest.toml

[1520ce14] + AbstractTrees v0.3.2

[944b1d66] + CodecZlib v0.6.0

[587475ba] + Flux v0.8.3

[e5e0dc1b] + Juno v0.8.0

[e89f7d12] + Media v0.5.0

[9f7883ad] + Tracker v0.2.6

[3bb67fe8] + TranscodingStreams v0.9.5

[a5390f91] + ZipFile v0.9.1

[9abbd945] + Profile

Building CodecZlib → ~/.julia/packages/CodecZlib/5t9zO/deps/build.log

julia>

25

Flux does not support the latest version of Colors which you have installed, so it downgrades to a version of Flux that doesn’t use Colors. CompatHelper: bump compat for "Colors" to "0.11" by github-actions[bot] · Pull Request #1038 · FluxML/Flux.jl · GitHub will fix that.

1 Like
26

Awsome. Thank you. It seems the protocol is “get rid of all packages, install flux, then . instal everything else.” (I’m not fully conversant with the dependency management in Julia, but I’m getting there :slight_smile: )