Porting an example from QuantEcon.jl to POMDPs.jl

Specific Domains
1

Just for fun (and to learn about POMDP) I am porting an example of dynamic programming from QuantEcon.jl to POMDPs.jl. The example is available here: https://lectures.quantecon.org/jl/discrete_dp.html#example-a-growth-model

My code for the implementation is:

using POMDPs
using POMDPModels
using POMDPModelTools
using DiscreteValueIteration

struct SimpleOG{TI <: Integer, T <: Real, TR <: AbstractArray{T}, TQ <: AbstractArray{T}} <: MDP{T, T}
    B :: TI
    M :: TI
    α :: T
    β :: T
    R :: TR
    Q :: TQ
end

function SimpleOG(; B::Integer = 10, M::Integer = 5, α::T = 0.5, β::T = 0.90) where {T <: Real}

    u(c) = c^α # utility function
    n = B + M + 1
    m = M + 1

    R = Matrix{T}(undef, n, m)
    Q = zeros(Float64, n, m, n)

    for a in 0:M
        Q[:, a + 1, (a:(a + B)) .+ 1] .= 1 / (B + 1)
        for s in 0:(B + M)
            R[s + 1, a + 1] = a <= s ? u(s - a) : -Inf
        end
    end

    return SimpleOG(B, M, α, β, R, Q)
end

POMDPs.states(simpleog::SimpleOG) = collect(0:(simpleog.M + simpleog.B))
POMDPs.n_states(simpleog::SimpleOG) = simpleog.B + simpleog.M + 1
POMDPs.stateindex(simpleog::SimpleOG, s) = Int(s) + 1

POMDPs.actions(simpleog::SimpleOG) = collect(0:simpleog.M)
POMDPs.n_actions(simpleog::SimpleOG) = simpleog.M + 1
POMDPs.actionindex(simpleog::SimpleOG, a) = Int(a) + 1

POMDPs.transition(simpleog::SimpleOG, s, a) = simpleog.Q[Int(a) + 1, Int(s) + 1, :]

POMDPs.reward(simpleog::SimpleOG, s, a) = simpleog.R[Int(s), Int(a)]
POMDPs.reward(simpleog::SimpleOG, s, a, sp) = reward(simpleog, s, a)

POMDPs.discount(simpleog::SimpleOG) = simpleog.β

g = SimpleOG()

When I try to solve the model I get:

julia> POMDPs.solve(g)
MethodError: no method matching solve(::SimpleOG{Int64,Float64,Array{Float64,2},Array{Float64,3}})
Closest candidates are:
  solve(!Matched::POMDPPolicies.FunctionSolver, !Matched::Union{MDP, POMDP}) at /Users/amrods/.julia/packages/POMDPPolicies/oW6ud/src/function.jl:23
  solve(!Matched::POMDPPolicies.RandomSolver, !Matched::Union{MDP, POMDP}) at /Users/amrods/.julia/packages/POMDPPolicies/oW6ud/src/random.jl:36
  solve(!Matched::POMDPPolicies.VectorSolver{A}, !Matched::MDP{S,A}) where {S, A} at /Users/amrods/.julia/packages/POMDPPolicies/oW6ud/src/vector.jl:23
  ...

Stacktrace:
 [1] top-level scope at In[14]:1

Can you help?

2

Hi @amrods, I see this question is now quite old - I apologize for missing it.

solve takes two arguments, a solver and an MDP or POMDP, so you should use something like solve(ValueIterationSolver(), g)

1 Like