# How to solve a Markov decision process with randomness

**URL:** <https://discourse.julialang.org/t/how-to-solve-a-markov-decision-process-with-randomness/96426>\
**Category:** Modelling & Simulations\
**Tags:** markov-decision, rl\
**Created:** [March 22, 2023, 12:24am UTC](https://discourse.julialang.org/t/how-to-solve-a-markov-decision-process-with-randomness/96426 "2023-03-22T00:24:21Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jian\_ZUO](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jian_zuo/32/33738_2.png) [@Jian\_ZUO](https://discourse.julialang.org/u/Jian_ZUO)\
**Post date:** [March 22, 2023, 12:24am UTC](https://discourse.julialang.org/t/how-to-solve-a-markov-decision-process-with-randomness/96426/1 "2023-03-22T00:24:21Z")

</div>

Hi,  
I have a Markov decision process as follows:  
states: S1, S2; the increment of S1 follows a Gamma law, i.e. S1(t+d)-S1(t) ~ gamma(alpha, beta), S1 has a fixed range of zero to a failure threshold FT (thus the **transition of S1 is stochastic** );  
S2 is a discrete variable with an increment step of d (d\>0), i.e., S2=0, d, 2d, …; the value fo S2 is lined to S1, every time S1 survives ( **value lower than FT** ) one step (d), the value of S2 increased d.

The action takes 0, 1, stands for two discrete actions on the system;  
The objective is to select a sequence of actions to minimize an operation cost (related to reward).

Is there a suitable package in Julia to solve such MDP (like value iteration)?  
Sorry, I just start trying to solve it in Julia, so I did not have any minimal code yet.  
Thanks!