LoadError: Can't differentiate loopinfo expression

When trying to run the code: (from an example)

using DeepQLearning
using POMDPs
using Flux
using POMDPModels
using POMDPSimulators
using POMDPPolicies

# load MDP model from POMDPModels or define your own!
mdp = SimpleGridWorld();

# Define the Q network (see Flux.jl documentation)
# the gridworld state is represented by a 2 dimensional vector.
model = Chain(Dense(2, 32), Dense(32, length(POMDPs.actions(mdp))))

exploration = EpsGreedyPolicy(mdp, LinearDecaySchedule(start=1.0, stop=0.01, steps=10000/2))

solver = DeepQLearningSolver(qnetwork = model, max_steps=10000, 
                             exploration_policy = exploration,
                             learning_rate=0.005,log_freq=500,
                             recurrence=false,double_q=true, dueling=true, prioritized_replay=true)
policy = solve(solver, mdp)

sim = RolloutSimulator(max_steps=30)
r_tot = simulate(sim, mdp, policy)
println("Total discounted reward for 1 simulation: $r_tot")

I am getting the following error:

ERROR: LoadError: Can't differentiate loopinfo expression
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] macro expansion
    @ .\simdloop.jl:79 [inlined]
  [3] Pullback
    @ .\reduce.jl:243 [inlined]
  [4] (::typeof(∂(mapreduce_impl)))(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface2.jl:0
  [5] Pullback
    @ .\reduce.jl:257 [inlined]
  [6] (::typeof(∂(mapreduce_impl)))(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface2.jl:0
  [7] Pullback
    @ .\reduce.jl:415 [inlined]
  [8] (::typeof(∂(_mapreduce)))(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface2.jl:0
  [9] Pullback
    @ .\reducedim.jl:318 [inlined]
 [10] Pullback (repeats 2 times)
    @ .\reducedim.jl:310 [inlined]
 [11] (::typeof(∂(mapreduce)))(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface2.jl:0
 [12] Pullback
    @ .\reducedim.jl:878 [inlined]
 [13] (::typeof(∂(#_sum#682)))(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface2.jl:0
 [14] Pullback
    @ .\reducedim.jl:878 [inlined]
 [15] (::typeof(∂(_sum)))(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface2.jl:0
 [16] Pullback (repeats 2 times)
    @ .\reducedim.jl:874 [inlined]
 [17] (::typeof(∂(sum)))(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface2.jl:0
 [18] Pullback
    @ C:\Users\Tyler\.julia\packages\DeepQLearning\Uet74\src\solver.jl:223 [inlined]
 [19] (::typeof(∂(λ)))(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface2.jl:0
 [20] (::Zygote.var"#54#55"{Zygote.Params, typeof(∂(λ)), Zygote.Context})(Δ::Float32)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface.jl:172
 [21] gradient(f::Function, args::Zygote.Params)
    @ Zygote C:\Users\Tyler\.julia\packages\Zygote\bRa8J\src\compiler\interface.jl:49
 [22] batch_train!(solver::DeepQLearningSolver{EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, NTuple{4, Symbol}}}, env::POMDPModelTools.MDPCommonRLEnv{AbstractArray{Float32, N} where N, SimpleGridWorld, StaticArrays.SVector{2, Int64}}, policy::NNPolicy{SimpleGridWorld, DeepQLearning.DuelingNetwork, Symbol}, optimizer::ADAM, target_q::DeepQLearning.DuelingNetwork, replay::PrioritizedReplayBuffer{Int32, Float32, CartesianIndex{2}, StaticArrays.SVector{2, Float32}, Matrix{Float32}}; discount::Float64)
    @ DeepQLearning C:\Users\Tyler\.julia\packages\DeepQLearning\Uet74\src\solver.jl:219
 [23] batch_train!
    @ C:\Users\Tyler\.julia\packages\DeepQLearning\Uet74\src\solver.jl:200 [inlined]
 [24] dqn_train!(solver::DeepQLearningSolver{EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, NTuple{4, Symbol}}}, env::POMDPModelTools.MDPCommonRLEnv{AbstractArray{Float32, N} where N, SimpleGridWorld, StaticArrays.SVector{2, Int64}}, policy::NNPolicy{SimpleGridWorld, DeepQLearning.DuelingNetwork, Symbol}, replay::PrioritizedReplayBuffer{Int32, Float32, CartesianIndex{2}, StaticArrays.SVector{2, Float32}, Matrix{Float32}})
    @ DeepQLearning C:\Users\Tyler\.julia\packages\DeepQLearning\Uet74\src\solver.jl:138
 [25] solve(solver::DeepQLearningSolver{EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, NTuple{4, Symbol}}}, env::POMDPModelTools.MDPCommonRLEnv{AbstractArray{Float32, N} where N, SimpleGridWorld, StaticArrays.SVector{2, Int64}})
    @ DeepQLearning C:\Users\Tyler\.julia\packages\DeepQLearning\Uet74\src\solver.jl:56
 [26] solve(solver::DeepQLearningSolver{EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, NTuple{4, Symbol}}}, problem::SimpleGridWorld)
    @ DeepQLearning C:\Users\Tyler\.julia\packages\DeepQLearning\Uet74\src\solver.jl:32
 [27] top-level scope
    @ C:\Users\Tyler\Documents\Julia\Learning\GridWorldMDP.jl:21
in expression starting at C:\Users\Tyler\Documents\Julia\Learning\GridWorldMDP.jl:21

This error occurs on windows (note I also tried on 1.6.2 with same result):

Julia Version 1.6.3
Commit ae8452a9e0 (2021-09-23 17:34 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, haswell)

and Linux:

Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 11th Gen Intel(R) Core(TM) i7-11700KF @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, icelake-client)

Possibly its an issue with SIMD in Intel x86_64 arch? I am not sure, nothing looks obvious wrong with the code. Any idea what could be causing this?

If someone can try it on a different arch that would be great, that is my best guess.

Running your program results in

  500 / 10000 eps 0.901 |  avgR 0.900 | Loss 1.521e+00 | Grad 1.188e+00 | EvalR -Inf 
Evaluation ... Avg Reward 3.69 | Avg Step 53.55 
 1000 / 10000 eps 0.802 |  avgR 0.917 | Loss 6.995e-01 | Grad 3.344e-01 | EvalR 3.690 
Evaluation ... Avg Reward 3.88 | Avg Step 56.39 
 1500 / 10000 eps 0.703 |  avgR 2.762 | Loss 9.779e-01 | Grad 6.094e-01 | EvalR 3.880 
Evaluation ... Avg Reward 1.66 | Avg Step 55.07 
 2000 / 10000 eps 0.604 |  avgR 2.621 | Loss 7.879e-01 | Grad 3.524e-01 | EvalR 1.660 
Evaluation ... Avg Reward 3.88 | Avg Step 52.42 
 2500 / 10000 eps 0.505 |  avgR 2.657 | Loss 8.044e-01 | Grad 1.252e+00 | EvalR 3.880 
Evaluation ... Avg Reward -0.60 | Avg Step 62.15 
 3000 / 10000 eps 0.406 |  avgR 2.565 | Loss 6.603e-01 | Grad 8.019e-01 | EvalR -0.600 
Evaluation ... Avg Reward 0.51 | Avg Step 54.33 
Saving new model with eval reward 0.510
 3500 / 10000 eps 0.307 |  avgR 2.828 | Loss 2.877e-01 | Grad 3.039e-01 | EvalR 0.510 
Evaluation ... Avg Reward -1.38 | Avg Step 62.52 
 4000 / 10000 eps 0.208 |  avgR 3.049 | Loss 6.996e-01 | Grad 2.613e-01 | EvalR -1.380 
Evaluation ... Avg Reward 1.89 | Avg Step 62.98 
 4500 / 10000 eps 0.109 |  avgR 4.049 | Loss 8.591e-01 | Grad 4.381e-01 | EvalR 1.890 
Evaluation ... Avg Reward -1.76 | Avg Step 49.11 
 5000 / 10000 eps 0.010 |  avgR 3.216 | Loss 4.404e-01 | Grad 4.823e-01 | EvalR -1.760 
Evaluation ... Avg Reward 1.70 | Avg Step 75.05 
 5500 / 10000 eps 0.010 |  avgR 3.392 | Loss 3.977e-01 | Grad 3.268e-01 | EvalR 1.700 
Evaluation ... Avg Reward 2.66 | Avg Step 64.54 
 6000 / 10000 eps 0.010 |  avgR 3.392 | Loss 2.256e-01 | Grad 6.759e-01 | EvalR 2.660 
Evaluation ... Avg Reward 3.20 | Avg Step 63.00 
Saving new model with eval reward 3.200
 6500 / 10000 eps 0.010 |  avgR 3.431 | Loss 3.875e-01 | Grad 9.832e-01 | EvalR 3.200 
Evaluation ... Avg Reward 3.22 | Avg Step 58.48 
 7000 / 10000 eps 0.010 |  avgR 3.706 | Loss 5.200e-01 | Grad 4.182e-01 | EvalR 3.220 
Evaluation ... Avg Reward 4.61 | Avg Step 51.44 
 7500 / 10000 eps 0.010 |  avgR 4.108 | Loss 8.112e-01 | Grad 3.001e-01 | EvalR 4.610 
Evaluation ... Avg Reward 5.45 | Avg Step 40.97 
 8000 / 10000 eps 0.010 |  avgR 3.922 | Loss 3.702e-01 | Grad 2.678e-01 | EvalR 5.450 
Evaluation ... Avg Reward 5.28 | Avg Step 42.95
 8500 / 10000 eps 0.010 |  avgR 3.598 | Loss 4.120e-01 | Grad 6.611e-01 | EvalR 5.280
Evaluation ... Avg Reward 2.86 | Avg Step 46.36
 9000 / 10000 eps 0.010 |  avgR 3.461 | Loss 3.909e-01 | Grad 4.633e-01 | EvalR 2.860
Evaluation ... Avg Reward 1.54 | Avg Step 61.40
 9500 / 10000 eps 0.010 |  avgR 3.578 | Loss 4.062e-01 | Grad 2.624e-01 | EvalR 1.540
Evaluation ... Avg Reward 1.30 | Avg Step 73.63
10000 / 10000 eps 0.010 |  avgR 3.353 | Loss 6.445e-02 | Grad 1.368e-01 | EvalR 1.300
Restore model with eval reward 3.200
Total discounted reward for 1 simulation: 3.0

on my platform

Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)        
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 6

I wonder if there are breaking changes between 1.6.1 and 1.6.2 and how to detect them.

Thanks for checking. I will switch to 1.6.1 on both systems and see if it works. Will report back soon.

I am getting the same error in 1.6.1:

Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 11th Gen Intel(R) Core(TM) i7-11700KF @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, icelake-client)

Here are my package versions:

julia> Pkg.status(["DeepQLearning", "POMDPs", "Flux", "POMDPModels", "POMDPSimulators", "POMDPPolicies"])
      Status ...
  [de0a67f4] DeepQLearning v0.6.4
  [587475ba] Flux v0.12.7
  [355abbd5] POMDPModels v0.4.14
  [182e52fb] POMDPPolicies v0.4.1
  [e0d0a172] POMDPSimulators v0.3.13
  [a93abf59] POMDPs v0.9.3

and

julia> Pkg.status(["Zygote"])
      Status ...
  [e88e6eb3] Zygote v0.6.17
julia> Pkg.status(["Zygote"])
      Status `~/.julia/environments/v1.6/Project.toml`
  [e88e6eb3] Zygote v0.5.2

julia> Pkg.status(["DeepQLearning", "POMDPs", "Flux", "POMDPModels", "POMDPSimulators", "POMDPPolicies"])
      Status `~/.julia/environments/v1.6/Project.toml`
  [de0a67f4] DeepQLearning v0.6.4
  [587475ba] Flux v0.11.3
  [355abbd5] POMDPModels v0.4.9
  [182e52fb] POMDPPolicies v0.4.1
  [e0d0a172] POMDPSimulators v0.3.12
  [a93abf59] POMDPs v0.9.3

Mine are different. I will change to yours and see what happens.

Since the last package in the error stack is Zygote, I am starting there.

After calling

@v1.6) pkg> update Zygote
    Updating registry at `~/.julia/registries/General`
   Installed ArrayInterface ─ v3.1.33
  Downloaded artifact: ImageMagick
  No Changes to `~/.julia/environments/v1.6/Project.toml`
    Updating `~/.julia/environments/v1.6/Manifest.toml`
  [4fba245c] ↑ ArrayInterface v3.1.32 ⇒ v3.1.33
  [082447d4] ↑ ChainRules v0.7.27 ⇒ v0.7.70
  [34da2185] ↑ Compat v2.2.1 ⇒ v3.39.0
  [b4f34e82] ↑ Distances v0.8.2 ⇒ v0.9.2
  [5789e2e9] ↑ FileIO v1.5.1 ⇒ v1.11.1
  [f6369f11] ↑ ForwardDiff v0.10.19 ⇒ v0.10.20
  [a2bd30eb] ↑ Graphics v1.1.0 ⇒ v1.1.1
  [cd3eb016] ↑ HTTP v0.8.19 ⇒ v0.9.16
  [872c559c] ↑ NNlib v0.7.11 ⇒ v0.7.29
  [b8a86587] ↑ NearestNeighbors v0.4.4 ⇒ v0.4.9
  [69de0a69] ↑ Parsers v2.0.4 ⇒ v2.0.5
  [3349acd9] ↑ ProtoBuf v0.11.2 ⇒ v0.11.3
  [c84ed2f1] ↑ Ratios v0.4.0 ⇒ v0.4.2
  [ae029012] ↑ Requires v0.5.2 ⇒ v1.1.3
  [bd369af6] ↑ Tables v1.5.2 ⇒ v1.6.0
  [9ac5f52a] + Tectonic v0.6.1
  [37f6aa50] ↑ TikzPictures v3.2.0 ⇒ v3.4.1
  [5c2747f8] + URIs v1.3.0
  [700de1a5] ↑ ZygoteRules v0.2.1 ⇒ v0.2.2
  [c73af94c] ↑ ImageMagick_jll v6.9.10-12+3 ⇒ v6.9.12+0
Precompiling project...
  64 dependencies successfully precompiled in 38 seconds (166 already precompiled)

julia> Pkg.status(["Zygote"])
      Status `~/.julia/environments/v1.6/Project.toml`
  [e88e6eb3] Zygote v0.5.2

Seems my Zygote packege is still behind. Is there something else I need to call to get to your version? Or is something broken if I cannot update Zygote?

Looks like you are in dependency hell;)

Maybe some help can be found here: How to find package preventing others from updating - #3 by Ralph_Smith

Pkg.add(name="Zygote", version="0.6.17")

should tell you, which package is preventing the update.

Looks like you are in dependency hell;)

Pkg.add(name="Zygote", version="0.6.17")

should tell you, which package is preventing the update.

You are correct. On one of my devices, this worked. On the other, it told me which package was preventing it from updating (ChainRules), however I was not able to delete that package. I think it was because my manifest.toml was messed up. So I just deleted the entire .julia folder, reinstalled all the packages with Pkg.add (or ]add ), and it seems to be working now. Thank you!

For someone investigating the original bug, seems to be an error in Zygote that was fixed at some point. Update Zygote.