Troubleshooting compatibility issues (preventing RL from downgrading Metal to 0.2)

joelreymont · February 23, 2024, 3:19pm

I find that installing ReinforcementLearning downgrades my Metal to version 0.2. This cuts my ability to run Lux on the Apple Silicon GPU, for example. I’m new to Julia and haven’t been able to figure out who requires Metal 0.2 and how to fix it.

What’s the best way to get my Metal version back up?

I tried forcing the issue with ] add Metal@1.0.0 and got this lengthy output that I can’t interpret.

Help!

(@v1.10) pkg> add Metal @1.0.0
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ReinforcementLearning [158674fc]:
 ReinforcementLearning [158674fc] log:
 ├─possible versions are: 0.2.0-0.10.2 or uninstalled
 ├─restricted to versions * by an explicit requirement, leaving only versions: 0.2.0-0.10.2
 ├─restricted by compatibility requirements with Requires [ae029012] to versions: 0.4.0-0.10.2 or uninstalled, leaving only versions: 0.4.0-0.10.2
 │ └─Requires [ae029012] log:
 │   ├─possible versions are: 0.5.0-1.3.0 or uninstalled
 │   └─restricted by compatibility requirements with Metal [dde4c033] to versions: 1.0.0-1.3.0
 │     └─Metal [dde4c033] log:
 │       ├─possible versions are: 0.0.1-1.0.0 or uninstalled
 │       └─restricted to versions 1.0.0 by an explicit requirement, leaving only versions: 1.0.0
 ├─restricted by compatibility requirements with Reexport [189a3867] to versions: [0.2.0-0.3.0, 0.9.0-0.10.2] or uninstalled, leaving only versions: 0.9.0-0.10.2
 │ └─Reexport [189a3867] log:
 │   ├─possible versions are: 0.2.0-1.2.2 or uninstalled
 │   └─restricted by compatibility requirements with Metal [dde4c033] to versions: 1.0.0-1.2.2
 │     └─Metal [dde4c033] log: see above
 └─restricted by compatibility requirements with ReinforcementLearningZoo [d607f57d] to versions: 0.2.0-0.6.0 or uninstalled — no versions left
   └─ReinforcementLearningZoo [d607f57d] log:
     ├─possible versions are: 0.1.0-0.8.3 or uninstalled
     ├─restricted by compatibility requirements with NNlib [872c559c] to versions: [0.1.0-0.5.12, 0.8.1-0.8.3] or uninstalled
     │ └─NNlib [872c559c] log:
     │   ├─possible versions are: 0.3.2-0.9.12 or uninstalled
     │   ├─restricted by compatibility requirements with Flux [587475ba] to versions: [0.3.2-0.8.21, 0.9.1-0.9.12]
     │   │ └─Flux [587475ba] log:
     │   │   ├─possible versions are: 0.4.1-0.14.12 or uninstalled
     │   │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.4.1-0.14.12
     │   │   ├─restricted by compatibility requirements with Metal [dde4c033] to versions: [0.4.1-0.13.16, 0.14.11-0.14.12] or uninstalled, leaving only versions: [0.4.1-0.13.16, 0.14.11-0.14.12]
     │   │   │ └─Metal [dde4c033] log: see above
     │   │   └─restricted by compatibility requirements with Adapt [79e6a3ab] to versions: 0.14.11-0.14.12 or uninstalled, leaving only versions: 0.14.11-0.14.12
     │   │     └─Adapt [79e6a3ab] log:
     │   │       ├─possible versions are: 0.3.0-4.0.1 or uninstalled
     │   │       └─restricted by compatibility requirements with Metal [dde4c033] to versions: 4.0.0-4.0.1
     │   │         └─Metal [dde4c033] log: see above
     │   ├─restricted by compatibility requirements with Adapt [79e6a3ab] to versions: [0.3.2-0.7.17, 0.9.10-0.9.12] or uninstalled, leaving only versions: [0.3.2-0.7.17, 0.9.10-0.9.12]
     │   │ └─Adapt [79e6a3ab] log: see above
     │   ├─restricted by compatibility requirements with ChainRulesCore [d360d2e6] to versions: [0.3.2-0.7.9, 0.7.27-0.9.12] or uninstalled, leaving only versions: [0.3.2-0.7.9, 0.9.10-0.9.12]
     │   │ └─ChainRulesCore [d360d2e6] log:
     │   │   ├─possible versions are: 0.1.0-1.22.1 or uninstalled
     │   │   └─restricted by compatibility requirements with Lux [b2108857] to versions: 1.0.0-1.22.1
     │   │     └─Lux [b2108857] log:
     │   │       ├─possible versions are: 0.3.0-0.5.17 or uninstalled
     │   │       ├─restricted to versions * by an explicit requirement, leaving only versions: 0.3.0-0.5.17
     │   │       ├─restricted by compatibility requirements with Metal [dde4c033] to versions: [0.3.0-0.5.13, 0.5.15-0.5.17] or uninstalled, leaving only versions: [0.3.0-0.5.13, 0.5.15-0.5.17]
     │   │       │ └─Metal [dde4c033] log: see above
     │   │       └─restricted by compatibility requirements with Adapt [79e6a3ab] to versions: [0.5.0-0.5.2, 0.5.14-0.5.17] or uninstalled, leaving only versions: [0.5.0-0.5.2, 0.5.15-0.5.17]
     │   │         └─Adapt [79e6a3ab] log: see above
     │   ├─restricted by compatibility requirements with Requires [ae029012] to versions: 0.6.0-0.9.12 or uninstalled, leaving only versions: [0.6.0-0.7.9, 0.9.10-0.9.12]
     │   │ └─Requires [ae029012] log: see above
     │   └─restricted by compatibility requirements with LoopVectorization [bdcacae8] to versions: [0.3.2-0.7.1, 0.7.4-0.9.12] or uninstalled, leaving only versions: [0.6.0-0.7.1, 0.7.4-0.7.9, 0.9.10-0.9.12]
     │     └─LoopVectorization [bdcacae8] log:
     │       ├─possible versions are: 0.1.0-0.12.166 or uninstalled
     │       ├─restricted by julia compatibility requirements to versions: 0.9.0-0.12.166 or uninstalled
     │       ├─restricted by compatibility requirements with ChainRulesCore [d360d2e6] to versions: [0.1.0-0.12.29, 0.12.31-0.12.166] or uninstalled, leaving only versions: [0.9.0-0.12.29, 0.12.31-0.12.166] or uninstalled
     │       │ └─ChainRulesCore [d360d2e6] log: see above
     │       └─restricted by compatibility requirements with VectorizationBase [3d5dd08c] to versions: 0.12.19-0.12.166 or uninstalled, leaving only versions: [0.12.19-0.12.29, 0.12.31-0.12.166] or uninstalled
     │         └─VectorizationBase [3d5dd08c] log:
     │           ├─possible versions are: 0.1.0-0.21.65 or uninstalled
     │           └─restricted by julia compatibility requirements to versions: 0.20.8-0.21.65 or uninstalled
     └─restricted by compatibility requirements with CUDA [052768ef] to versions: 0.1.0-0.1.4 or uninstalled
       └─CUDA [052768ef] log:
         ├─possible versions are: 0.1.0-5.2.0 or uninstalled
         ├─restricted by julia compatibility requirements to versions: [2.3.0, 2.5.0-5.2.0] or uninstalled
         └─restricted by compatibility requirements with Adapt [79e6a3ab] to versions: 5.2.0 or uninstalled
           └─Adapt [79e6a3ab] log: see above

joelreymont · February 23, 2024, 5:11pm

Apparently, it’s ReinforcementLearningCore that’s causing the trouble.

It seems to be pointing at Adapt but that package does not depend on Metal at all.

(@v1.10) pkg> add Metal@1.0.0
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ReinforcementLearningCore [de1b191a]:
 ReinforcementLearningCore [de1b191a] log:
 ├─possible versions are: 0.1.0-0.13.0 or uninstalled
 ├─restricted to versions * by an explicit requirement, leaving only versions: 0.1.0-0.13.0
 ├─restricted by compatibility requirements with Adapt [79e6a3ab] to versions: 0.1.0-0.2.0 or uninstalled, leaving only versions: 0.1.0-0.2.0
 │ └─Adapt [79e6a3ab] log:
 │   ├─possible versions are: 0.3.0-4.0.1 or uninstalled
 │   └─restricted by compatibility requirements with Metal [dde4c033] to versions: 4.0.0-4.0.1
 │     └─Metal [dde4c033] log:
 │       ├─possible versions are: 0.0.1-1.0.0 or uninstalled
 │       └─restricted to versions 1.0.0 by an explicit requirement, leaving only versions: 1.0.0
 ├─restricted by compatibility requirements with CuArrays [3a865a2d] to versions: [0.2.0, 0.4.0-0.13.0] or uninstalled, leaving only versions: 0.2.0
 │ └─CuArrays [3a865a2d] log:
 │   ├─possible versions are: 0.2.1-2.2.2 or uninstalled
 │   └─restricted by julia compatibility requirements to versions: uninstalled
 └─restricted by compatibility requirements with Distributions [31c24e10] to versions: 0.8.0-0.13.0 or uninstalled — no versions left
   └─Distributions [31c24e10] log:
     ├─possible versions are: 0.16.0-0.25.107 or uninstalled
     ├─restricted by compatibility requirements with ReinforcementLearningCore [de1b191a] to versions: 0.22.0-0.25.107
     │ └─ReinforcementLearningCore [de1b191a] log: see above
     └─restricted by compatibility requirements with FillArrays [1a297f60] to versions: [0.16.0-0.21.9, 0.25.49-0.25.107] or uninstalled, leaving only versions: 0.25.49-0.25.107
       └─FillArrays [1a297f60] log:
         ├─possible versions are: 0.2.0-1.9.3 or uninstalled
         ├─restricted by compatibility requirements with Lux [b2108857] to versions: 0.13.0-1.9.3 or uninstalled
         │ └─Lux [b2108857] log:
         │   ├─possible versions are: 0.3.0-0.5.17 or uninstalled
         │   ├─restricted to versions * by an explicit requirement, leaving only versions: 0.3.0-0.5.17
         │   ├─restricted by compatibility requirements with Metal [dde4c033] to versions: [0.3.0-0.5.13, 0.5.15-0.5.17] or uninstalled, leaving only versions: [0.3.0-0.5.13, 0.5.15-0.5.17]
         │   │ └─Metal [dde4c033] log: see above
         │   └─restricted by compatibility requirements with Adapt [79e6a3ab] to versions: [0.5.0-0.5.2, 0.5.14-0.5.17] or uninstalled, leaving only versions: [0.5.0-0.5.2, 0.5.15-0.5.17]
         │     └─Adapt [79e6a3ab] log: see above
         └─restricted by compatibility requirements with Distributions [31c24e10] to versions: 0.9.0-1.9.3, leaving only versions: 0.13.0-1.9.3
           └─Distributions [31c24e10] log: see above

mkitti · February 23, 2024, 8:07pm

We may need to modify the compat bounds of RL core. For your oen purposes, dev the package into a temporary environment, modify the local copy, and then dev the package (from the local copy) into the environment.

joelreymont · February 23, 2024, 8:32pm

Where are the bounds, though?

I cannot find Metal in the Project.toml of RL Core.

avikpal · February 25, 2024, 6:12pm

It is the Adapt compat. Seems like quite a few unmerged CompatHelper PRs Pull requests · JuliaReinforcementLearning/ReinforcementLearning.jl · GitHub

joelreymont · February 26, 2024, 4:22pm

I checked Adapt’s Project.toml and there’s no dependency on Metal.

What am I missing?

avikpal · February 26, 2024, 5:47pm

I meant the compat for Adapt in one of the packages needs to be bumped to 4

joelreymont · February 27, 2024, 8:58am

How do I tell the package manager to consider the dev version of a package when I upgrade other packages? For example, I have dev-d ReinforcementLearning

(@v1.10) pkg> status
Status `~/.julia/environments/v1.10/Project.toml`
  [6e4b80f9] BenchmarkTools v1.4.0
⌃ [b2108857] Lux v0.5.15
⌃ [dde4c033] Metal v0.5.0
  [158674fc] ReinforcementLearning v0.11.0-dev `~/.julia/dev/ReinforcementLearning`
  [295af30f] Revise v3.5.14
Info Packages marked with ⌃ have new versions available and may be upgradable.

and it’s using Adapt v4

diff --git a/src/ReinforcementLearningCore/Project.toml b/src/ReinforcementLearningCore/Project.toml
index b5096027..0258037d 100644
--- a/src/ReinforcementLearningCore/Project.toml
+++ b/src/ReinforcementLearningCore/Project.toml
@@ -29,7 +29,7 @@ cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"

 [compat]
 AbstractTrees = "0.3, 0.4"
-Adapt = "3"
+Adapt = "4"
 CUDA = "4, 5"
 ChainRulesCore = "1"
 CircularArrayBuffers = "0.1"

but when I try to upgrade a package it’s telling me that Adapt is restricted to v3

(@v1.10) pkg> add Metal@1
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Adapt [79e6a3ab]:
 Adapt [79e6a3ab] log:
 ├─possible versions are: 0.3.0-4.0.1 or uninstalled
 ├─restricted by compatibility requirements with Metal [dde4c033] to versions: 4.0.0-4.0.1
 │ └─Metal [dde4c033] log:
 │   ├─possible versions are: 0.0.1-1.0.0 or uninstalled
 │   └─restricted to versions 1 by an explicit requirement, leaving only versions: 1.0.0
 └─restricted by compatibility requirements with ReinforcementLearningCore [de1b191a] to versions: 3.0.0-3.7.2 — no versions left
   └─ReinforcementLearningCore [de1b191a] log:
     ├─possible versions are: 0.1.0-0.13.0 or uninstalled
     └─restricted to versions 0.9 by ReinforcementLearning [158674fc], leaving only versions: 0.9.0-0.9.4
       └─ReinforcementLearning [158674fc] log:
         ├─possible versions are: 0.11.0 or uninstalled
         └─ReinforcementLearning [158674fc] is fixed to version 0.11.0-dev

I can’t separately dev ReinforcementLearningCore

(@v1.10) pkg> dev ReinforcementLearningCore
Path `/Users/joelr/.julia/dev/ReinforcementLearning` exists and looks like the correct repo. Using existing path.
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ReinforcementLearningCore [de1b191a]:
 ReinforcementLearningCore [de1b191a] log:
 ├─possible versions are: 0.15.1 or uninstalled
 ├─restricted to versions 0.9 by ReinforcementLearning [158674fc] — no versions left
 │ └─ReinforcementLearning [158674fc] log:
 │   ├─possible versions are: 0.11.0 or uninstalled
 │   └─ReinforcementLearning [158674fc] is fixed to version 0.11.0-dev
 └─ReinforcementLearningCore [de1b191a] is fixed to version 0.15.1

but I see that’s is using Adapt v4 in the dev version

❯ pwd
/Users/joelr/.julia/dev/ReinforcementLearning
❯ grep Adapt src/ReinforcementLearningCore/Project.toml
Adapt = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
Adapt = "4"

joelreymont · February 28, 2024, 10:26am

I have fallen and I can’t get up! Please help!

ReinforcementLearning (RL) has switched to sub-packages under the src directory. Older versions of these same sub-packages are available as packages in the Julia general registry.

I checked out a local copy of RL for development

(@v1.10) pkg> st
Status `~/.julia/environments/v1.10/Project.toml`
⌃ [b2108857] Lux v0.5.15
⌃ [dde4c033] Metal v0.5.0
  [158674fc] ReinforcementLearning v0.11.0-dev `~/Work/Julia/ReinforcementLearning.jl`

and activated my local copy

(@v1.10) pkg> activate ~/Work/Julia/ReinforcementLearning.jl
  Activating project at `~/Work/Julia/ReinforcementLearning.jl`

I don’t know why the top RL Project.toml does not specify compatibility with the latest versions of its sub-packages. Perhaps, it doesn’t matter.

(ReinforcementLearning) pkg> st
Project ReinforcementLearning v0.11.0-dev
Status `~/Work/Julia/ReinforcementLearning.jl/Project.toml`
   [189a3867] Reexport v1.2.2
→⌅ [e575027e] ReinforcementLearningBase v0.11.1
→⌅ [de1b191a] ReinforcementLearningCore v0.9.4
→⌃ [dd1544ca] ReinforcementLearningDatasets v0.1.0
→⌃ [25e41dd2] ReinforcementLearningEnvironments v0.7.2
→⌅ [d607f57d] ReinforcementLearningZoo v0.6.0

The sub-packages do require the latest versions of their sibling packages so I need to dev the sibling package that each sub-package requires. There’s a script (devmode.jl, top level in RL) that simplifies the setup, e.g.

function activate_devmode!()
    @info "Switching to dev mode. You are now using your local versions of the RL.jl packages instead of the registered releases."
    #RLBase
    #No dependency to dev
    #RLCore
    Pkg.activate("src/ReinforcementLearningCore")
    Pkg.develop(path="src/ReinforcementLearningBase")
    #RLZoo
    Pkg.activate("src/ReinforcementLearningZoo")
    Pkg.develop(path="src/ReinforcementLearningCore")
    Pkg.develop(path="src/ReinforcementLearningBase")    
    #RLEnvironments
    Pkg.activate("src/ReinforcementLearningEnvironments")
    Pkg.develop(path="src/ReinforcementLearningBase")
    #RLExperiments
    Pkg.activate("src/ReinforcementLearningExperiments")
    Pkg.develop(path="src/ReinforcementLearningZoo")
    Pkg.develop(path="src/ReinforcementLearningEnvironments")
    Pkg.develop(path="src/ReinforcementLearningCore")
    Pkg.develop(path="src/ReinforcementLearningBase")
    #RL
    Pkg.activate(".")
    Pkg.develop(path="src/ReinforcementLearningZoo")
    Pkg.develop(path="src/ReinforcementLearningEnvironments")
    Pkg.develop(path="src/ReinforcementLearningCore")
    Pkg.develop(path="src/ReinforcementLearningBase")
end

It works until it gets to src/ReinforcementLearningExperiments and then

(ReinforcementLearningExperi...) pkg> dev src/ReinforcementLearningZoo
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ReinforcementLearningCore [de1b191a]:
 ReinforcementLearningCore [de1b191a] log:
 ├─possible versions are: 0.1.0-0.13.0 or uninstalled
 └─restricted to versions 0.15 by ReinforcementLearningExperiments [6bd458e5] — no versions left
   └─ReinforcementLearningExperiments [6bd458e5] log:
     ├─possible versions are: 0.5.0 or uninstalled
     └─ReinforcementLearningExperiments [6bd458e5] is fixed to version 0.5.0

The ReinforcementLearningCore sub-package is at version 0.15 whereas the one available in the Julia registry is at 0.13. The ReinforcementLearningExperiments sub-package is expecting the latest.

Maybe I can dev the ReinforcementLearningCore sub-package to resolve the above error?

(ReinforcementLearningExperi...) pkg> dev src/ReinforcementLearningCore
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ReinforcementLearningZoo [d607f57d]:
 ReinforcementLearningZoo [d607f57d] log:
 ├─possible versions are: 0.1.0-0.8.3 or uninstalled
 └─restricted to versions 0.10.1-0.10 by ReinforcementLearningExperiments [6bd458e5] — no versions left
   └─ReinforcementLearningExperiments [6bd458e5] log:
     ├─possible versions are: 0.5.0 or uninstalled
     └─ReinforcementLearningExperiments [6bd458e5] is fixed to version 0.5.0

The Zoo sub-package is at version 0.10 but the one in the Julia registry is at 0.8.3.

So old packages in the Julia registry interfere with the ones I’m trying to develop. How do I work around this problem?

Thank you, Joel

joelreymont · February 28, 2024, 11:19am

I may have snatched victory from the jaws of defeat by giving dev all the required sub-packages at the same time

(ReinforcementLearningExperi...) pkg>  dev src/ReinforcementLearningZoo src/ReinforcementLearningEnvironments src/ReinforcementLearningCore src/ReinforcementLearningBase
   Resolving package versions...
   Installed RelocatableFolders ─ v1.0.1
   Installed Libiconv_jll ─────── v1.17.0+0
   Installed StringEncodings ──── v0.3.7
   Installed Mustache ─────────── v1.0.19
   Installed Highlights ───────── v0.5.2
   Installed YAML ─────────────── v0.4.9
   Installed Weave ────────────── v0.10.12
  Downloaded artifact: Libiconv
    Updating `~/Work/Julia/ReinforcementLearning.jl/src/ReinforcementLearningExperiments/Project.toml`
⌃ [052768ef] + CUDA v5.1.2
  [31c24e10] + Distributions v0.25.107
...

joelreymont · March 1, 2024, 1:20pm

This PR should fix the problem.