# Help I am new: Is this MDP righ?

**URL:** <https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [March 26, 2022, 8:04pm UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522 "2022-03-26T20:04:08Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![clrescobar](https://avatars.discourse-cdn.com/v4/letter/c/a3d4f5/32.png) [@clrescobar](https://discourse.julialang.org/u/clrescobar)\
**Post date:** [March 26, 2022, 8:04pm UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/1 "2022-03-26T20:04:08Z")

</div>

This post was temporarily hidden by the community for possibly being off-topic, unfocused, inappropriate, or spammy.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 26, 2022, 8:30pm UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/2 "2022-03-26T20:30:54Z")

</div>

Hi, first a comment on formatting code: you could enclose your complete MWE in triple back ticks and it would show as code in Discourse.

That said your own definition

```julia
struct MyMDP <: MDP{Int,Int} end

```

does indeed not contain a member `indices` which these functions

```julia
POMDPs.stateindex(m::MyMDP, s) = m.indices[s]
POMDPs.actionindex(m::MyMDP, a) = m.indices[a]

```

try to access. How did you get the idea this could work?

---

<div class="post-metadata">

**Author:** ![clrescobar](https://avatars.discourse-cdn.com/v4/letter/c/a3d4f5/32.png) [@clrescobar](https://discourse.julialang.org/u/clrescobar)\
**Post date:** [March 26, 2022, 9:54pm UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/3 "2022-03-26T21:54:28Z")

</div>

Hello,

First of all thank you so much for your quick response and sorry for the format.

I tried to follow the documentation. How can I define the member indices?

Thanks!

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 26, 2022, 10:06pm UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/4 "2022-03-26T22:06:18Z")

</div>

> [@clrescobar](#):
>
> I tried to follow the documentation.

Thanks. Your example looks a bit like [this one](https://juliapomdp.github.io/POMDPs.jl/latest/def_pomdp/#Object-oriented). I tried

```julia
struct MyMDP <: MDP{Int,Int} 
    indices::Dict{String, Int}
    MyMDP() = new(Dict{Int, Int}())
end

```

but it looks to me like you have to fill the index dictionary with some state action pairs like

```julia
Dict{Int, Int}(0=>? ...)

```

?

---

<div class="post-metadata">

**Author:** ![clrescobar](https://avatars.discourse-cdn.com/v4/letter/c/a3d4f5/32.png) [@clrescobar](https://discourse.julialang.org/u/clrescobar)\
**Post date:** [March 27, 2022, 9:36am UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/5 "2022-03-27T09:36:56Z")

</div>

Hi @goerch,

I’ve tried what you said:

> using POMDPs  
> using POMDPModelTools: Deterministic, Uniform, SparseCat  
> using TabularTDLearning  
> using POMDPPolicies  
> using POMDPModels  
> struct MyMDP \<: MDP{Int,Int}  
> indices::Dict{Int, Int}  
> MyMDP() = new(Dict(0=\>1,1=\>2,2=\>2,3=\>4))  
> end  
> mdp = MyMDP()  
> POMDPs.actions(m::MyMDP) = [0,1,2,3]  
> POMDPs.states(m::MyMDP) = [0,1,2,3]  
> POMDPs.discount(m::MyMDP) = 0.95  
> POMDPs.stateindex(m::MyMDP, s) = m.indices[s]  
> POMDPs.actionindex(m::MyMDP, a) = m.indices[a]  
> POMDPs.initialstate(m::MyMDP) = Uniform([0,1,2,3])  
> function POMDPs.transition(m::MyMDP, s, a)  
> if s == 0 && a == 0  
> return SparseCat([0,1,2,3], [1,0,0,0])  
> elseif s == 0 && a == 1  
> return SparseCat([0,1,2,3], [0.7, 0.3,0,0])  
> elseif s == 0 && a == 2  
> return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])  
> elseif s == 0 && a == 3  
> return SparseCat([0,1,2,3], [0,0.2,0.5,0.3])  
> elseif s == 1 && a == 0  
> return SparseCat([0,1,2,3], [0.7, 0.3,0,0])  
> elseif s == 1 && a == 1  
> return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])  
> elseif s == 1 && a == 2  
> return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])  
> elseif s == 2 && a == 0  
> return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])  
> elseif s == 2 && a == 1  
> return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])  
> elseif s == 3 && a == 0  
> return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])  
> end  
> end  
> function POMDPs.reward(m::MyMDP, s, a)  
> if s == 0 && a == 0  
> return -45  
> elseif s == 0 && a == 1  
> return -40  
> elseif s == 0 && a == 2  
> return -50  
> elseif s == 0 && a == 3  
> return -70  
> elseif s == 1 && a == 0  
> return -14  
> elseif s == 1 && a == 1  
> return -44  
> elseif s == 1 && a == 2  
> return -54  
> elseif s == 1 && a == 3  
> return -74  
> elseif s == 2 && a == 0  
> return -8  
> elseif s == 2 && a == 1  
> return -38  
> elseif s == 3 && a == 0  
> return -12  
> end  
> end  
> q\_learning\_solver = QLearningSolver(n\_episodes=10,  
> learning\_rate=0.8,  
> exploration\_policy=EpsGreedyPolicy(mdp, 0.5),  
> verbose=false);  
> q\_learning\_policy = solve(q\_learning\_solver, mdp);

But I get “ERROR: ArgumentError: Sampler for this object is not defined” for

> q\_learning\_policy = solve(q\_learning\_solver, mdp);

I have also tried

> exppolicy = EpsGreedyPolicy(mdp, 0.01)  
> solver = QLearningSolver(exppolicy, learning\_rate=0.1, n\_episodes=50, max\_episode\_length=50, eval\_every=50, n\_eval\_traj=100)  
> policy = solve(solver, mdp)

That I found [here](https://github.com/JuliaPOMDP/TabularTDLearning.jl) but in

> solver = QLearningSolver(exppolicy, learning\_rate=0.1, n\_episodes=50, max\_episode\_length=50, eval\_every=50, n\_eval\_traj=100)

It says: ERROR: MethodError: no method matching QLearningSolver(::EpsGreedyPolicy{POMDPPolicies.var"#17#18"{Float64}, Random.\_GLOBAL\_RNG, Vector{Int64}}; learning\_rate=0.1, n\_episodes=50, max\_episode\_length=50, eval\_every=50, n\_eval\_traj=100)

I do not undertand what happens. Thank you for your help.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 27, 2022, 11:17am UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/6 "2022-03-27T11:17:39Z")

</div>

I instrumented `transition` and `reward`

```julia
function POMDPs.transition(m::MyMDP, s, a)
    if s == 0 && a == 0
        return SparseCat([0,1,2,3], [1,0,0,0])
[...]
    else
        @assert false
    end    
end

function POMDPs.reward(m::MyMDP, s, a)
    if s == 0 && a == 0
        return -45
[...]
    else
        @assert false
    end        
end    

```

and get

```julia
ERROR: AssertionError: false

```

in `transition`. So it seems this function is incomplete?

---

<div class="post-metadata">

**Author:** ![clrescobar](https://avatars.discourse-cdn.com/v4/letter/c/a3d4f5/32.png) [@clrescobar](https://discourse.julialang.org/u/clrescobar)\
**Post date:** [March 27, 2022, 11:28am UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/7 "2022-03-27T11:28:05Z")

</div>

Hello,

I have also tried:

```julia
function POMDPs.transition(m::ClinicalTrialsMDP, s, a)
    if s == 0 && a == 0 
        return SparseCat([0,1,2,3], [1,0,0,0])
    elseif s == 0 && a == 1 
        return SparseCat([0,1,2,3], [0.7, 0.3,0,0])
    elseif s == 0 && a == 2 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
    elseif s == 0 && a == 3 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])

    elseif s == 1 && a == 0 
        return SparseCat([0,1,2,3], [0.7, 0.3,0,0])
    elseif s == 1 && a == 1 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
    elseif s == 1 && a == 2 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
    elseif s == 1 && a == 3 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
        
    elseif s == 2 && a == 0 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
    elseif s == 2 && a == 1 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 2 && a == 2 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 2 && a == 3 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])

    elseif s == 3 && a == 0 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 3 && a == 1 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 3 && a == 2 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 3 && a == 3 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    end
end 

function POMDPs.reward(m::ClinicalTrialsMDP, s, a)
    if s == 0 && a == 0 
        return -45
    elseif s == 0 && a == 1 
        return -40
    elseif s == 0 && a == 2 
        return -50
    elseif s == 0 && a == 3 
        return -70
    
    elseif s == 1 && a == 0 
        return -14
    elseif s == 1 && a == 1 
        return -44
    elseif s == 1 && a == 2 
        return -54
    elseif s == 1 && a == 3 
        return -74   

    elseif s == 2 && a == 0 
        return -8
    elseif s == 2 && a == 1 
        return -38
    
    elseif s == 3 && a == 0 
        return -12
    end
end 

```

But still not working.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 27, 2022, 11:48am UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/8 "2022-03-27T11:48:07Z")

</div>

The following variation

```julia
using POMDPs
using POMDPModelTools
using QuickPOMDPs: QuickPOMDP
using TabularTDLearning
using POMDPPolicies
using POMDPModels
using Parameters, Random

struct MyMDP <: MDP{Int,Int} 
    indices::Dict{Int, Int}
    MyMDP() = new(Dict{Int, Int}(0=>1,1=>2,2=>3,3=>4))
end

mdp = MyMDP()

POMDPs.actions(m::MyMDP) = [0,1,2,3]
POMDPs.states(m::MyMDP) = [0,1,2,3]
POMDPs.discount(m::MyMDP) = 0.95
POMDPs.stateindex(m::MyMDP, s) = m.indices[s]
POMDPs.actionindex(m::MyMDP, a) = m.indices[a]
POMDPs.initialstate(m::MyMDP) = Uniform([0,1,2,3])

function POMDPs.transition(m::MyMDP, s, a)
    if s == 0 && a == 0 
        return SparseCat([0,1,2,3], [1,0,0,0])
    elseif s == 0 && a == 1 
        return SparseCat([0,1,2,3], [0.7, 0.3,0,0])
    elseif s == 0 && a == 2 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
    elseif s == 0 && a == 3 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])

    elseif s == 1 && a == 0 
        return SparseCat([0,1,2,3], [0.7, 0.3,0,0])
    elseif s == 1 && a == 1 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
    elseif s == 1 && a == 2 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
    elseif s == 1 && a == 3 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
        
    elseif s == 2 && a == 0 
        return SparseCat([0,1,2,3], [0.2, 0.5,0.3,0])
    elseif s == 2 && a == 1 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 2 && a == 2 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 2 && a == 3 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])

    elseif s == 3 && a == 0 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 3 && a == 1 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 3 && a == 2 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    elseif s == 3 && a == 3 
        return SparseCat([0,1,2,3], [0, 0.2,0.5,0.3])
    else
        @show "transition", s, a
        return Uniform([0, 1, 2, 3])
    end    
end

function POMDPs.reward(m::MyMDP, s, a)
    if s == 0 && a == 0
        return -45
    elseif s == 0 && a == 1
        return -40
    elseif s == 0 && a == 2
        return -50
    elseif s == 0 && a == 3
        return -70
    elseif s == 1 && a == 0
        return -14
    elseif s == 1 && a == 1
        return -44
    elseif s == 1 && a == 2
        return -54
    elseif s == 1 && a == 3
        return -74  
    elseif s == 2 && a == 0
        return -8
    elseif s == 2 && a == 1
        return -38
    elseif s == 3 && a == 0
        return -12
    else
        @show "reward", s, a
        return 0
    end        
end    

q_learning_solver = QLearningSolver(n_episodes=10,
                                learning_rate=0.8,
                                exploration_policy=EpsGreedyPolicy(mdp, 0.5),
                                verbose=false);
q_learning_policy = solve(q_learning_solver, mdp);    

```

does something for me without showing errors. I don’t know if it is what you intended it to do.

---

<div class="post-metadata">

**Author:** ![clrescobar](https://avatars.discourse-cdn.com/v4/letter/c/a3d4f5/32.png) [@clrescobar](https://discourse.julialang.org/u/clrescobar)\
**Post date:** [March 27, 2022, 12:20pm UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/9 "2022-03-27T12:20:56Z")

</div>

Hello,

Thank you so much for your help! I t works but I need to interpret first the results. Can you explain me why you use in transition:

```julia
else
        @show "transition", s, a
        return Uniform([0, 1, 2, 3])
    end    

```

and in reward

```julia
else
        @show "reward", s, a
        return 0
    end   

```

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 27, 2022, 12:28pm UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/10 "2022-03-27T12:28:21Z")

</div>

> [@clrescobar](#):
>
> Can you explain me why you use in transition:

I only tried to make the functions well defined, i.e. returning some data in every case (which seems required). It at least showed that some cases in `reward` where not defined…

---

<div class="post-metadata">

**Author:** ![clrescobar](https://avatars.discourse-cdn.com/v4/letter/c/a3d4f5/32.png) [@clrescobar](https://discourse.julialang.org/u/clrescobar)\
**Post date:** [March 27, 2022, 2:17pm UTC](https://discourse.julialang.org/t/help-i-am-new-is-this-mdp-righ/78522/11 "2022-03-27T14:17:42Z")

</div>

Hello,

I understand, but doing like you did means that every transition is possible. What if I am in state 2 (s==2) and I can only take action 0 and 1 (a==0, a==1)?

If I had understand your code, when you do :

```julia
 @show "transition", s, a
        return Uniform([0, 1, 2, 3])
    end    
end

```

Every transition is possible, right?