# Continous Markov Processes

**URL:** <https://discourse.julialang.org/t/continous-markov-processes/40090>\
**Category:** New to Julia\
**Created:** [May 24, 2020, 7:11pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090 "2020-05-24T19:11:37Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [May 24, 2020, 7:11pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/1 "2020-05-24T19:11:37Z")

</div>

Hi,

I just came across the “ContinuousTimeMarkov” written by Tom Pappas. Unfortunately, there are no examples in the repository and no documentation that I could find.

Could somebody, possibly Tom, provide me with one or more examples of how to use the package (without all the @test macros, which confuse me.

Thank you,

Gordon

---

<div class="post-metadata">

**Author:** ![dpsanders](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dpsanders/32/3573_2.png) [@dpsanders](https://discourse.julialang.org/u/dpsanders)\
**Post date:** [May 24, 2020, 7:17pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/2 "2020-05-24T19:17:41Z")

</div>

Do you mean the package [GitHub - tpapp/ContinuousTimeMarkov.jl: Julia package for Markov chains in continuous time.](https://github.com/tpapp/ContinuousTimeMarkov.jl) by Tamas Papp?  
Please provide links when referring to packages.

When a package has no examples or docs, it usually means it’s designed to be work in progress that the author is not ready to release yet.

But in fact often the tests are the best way to see how to use the package.  
The macro `@test <statement>` just tests if `<statement>` is true or not.

---

<div class="post-metadata">

**Author:** ![dpsanders](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dpsanders/32/3573_2.png) [@dpsanders](https://discourse.julialang.org/u/dpsanders)\
**Post date:** [May 24, 2020, 7:19pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/3 "2020-05-24T19:19:00Z")

</div>

```julia
julia> using Test

julia> @test 1 == 1
Test Passed

julia> @test 1 == 2
Test Failed at REPL[3]:1
  Expression: 1 == 2
   Evaluated: 1 == 2
ERROR: There was an error during testing

```

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 24, 2020, 7:19pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/4 "2020-05-24T19:19:48Z")

</div>

Here’s a tutorial on continuous time Markov processes:

[https://diffeq.sciml.ai/latest/tutorials/discrete\_stochastic\_example/](https://diffeq.sciml.ai/latest/tutorials/discrete_stochastic_example/)

In the context of epidemiology, there’s a repo with a lot of nice examples ([GitHub - epirecipes/sir-julia: Various implementations of the classical SIR model in Julia](https://github.com/epirecipes/sir-julia)) and [https://github.com/epirecipes/sir-julia/blob/master/markdown/rn\_mtk/rn\_mtk.md](https://github.com/epirecipes/sir-julia/blob/master/markdown/rn_mtk/rn_mtk.md) is an example of running through the range of models from ODEs, SDEs, and SSAs. The DiffEq benchmarks have quite a few examples as well of large-scale SSAs like:

[https://benchmarks.sciml.ai/html/Jumps/Diffusion\_CTRW.html](https://benchmarks.sciml.ai/html/Jumps/Diffusion_CTRW.html)

We don’t have any examples on the newest method since it was just released, but we now have post-leap adaptive tau-leaping and regular-jump Euler-Maruyama for jump diffusions, so if that’s what you’re looking for that could be helpful too.

Another nice related package on this topic is Gillespie.jl:

> **[GitHub - sdwfrost/Gillespie.jl: Stochastic Gillespie-type simulations using...](https://github.com/sdwfrost/Gillespie.jl)**
>
> Stochastic Gillespie-type simulations using Julia. Contribute to sdwfrost/Gillespie.jl development by creating an account on GitHub.

We probably need to increase the discoverability of these tools.

---

<div class="post-metadata">

**Author:** ![erlebach1](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@erlebach1](https://discourse.julialang.org/u/erlebach1)\
**Post date:** [May 24, 2020, 8:16pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/5 "2020-05-24T20:16:23Z")

</div>

Thanks, Chris. I have discovered all of these in the past few days and am in the process of getting some of the Julia code in epirecipe to run. They were written for Julia 0.6 I believe. Biosimulation.jl is also a potentially useful package.

By the way, are jump processes the same thing as Continuous Time Markov Chains?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 24, 2020, 8:21pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/6 "2020-05-24T20:21:36Z")

</div>

> [@erlebach1](#):
>
> By the way, are jump processes the same thing as Continuous Time Markov Chains?

Yes, jump processes are continuous-time Markov chains, and Gillespie’s SSA algorithm is one classic method for simulating them.

> [@erlebach1](#):
>
> Thanks, Chris. I have discovered all of these in the past few days and am in the process of getting some of the Julia code in epirecipe to run. They were written for Julia 0.6 I believe. Biosimulation.jl is also a potentially useful package.

Which website? The refreshed one should be good since it was just made. The older epirecipes were made for v0.6 but should still work (if you add `using ParameterizedFunctions`)

---

<div class="post-metadata">

**Author:** ![erlebach1](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@erlebach1](https://discourse.julialang.org/u/erlebach1)\
**Post date:** [May 24, 2020, 9:18pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/7 "2020-05-24T21:18:35Z")

</div>

Which website?

[http://epirecip.es/epicookbook/](http://epirecip.es/epicookbook/)

Thanks for your links,

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 24, 2020, 9:39pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/8 "2020-05-24T21:39:59Z")

</div>

Simon’s newer version is here: [https://github.com/epirecipes/sir-julia](https://github.com/epirecipes/sir-julia)

---

<div class="post-metadata">

**Author:** ![erlebach1](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@erlebach1](https://discourse.julialang.org/u/erlebach1)\
**Post date:** [May 24, 2020, 9:58pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/9 "2020-05-24T21:58:07Z")

</div>

Thanks. On the original site, there are a few network programs in R. You would not happen to know of any Julia implementations?

Have you ever worked with the Household-Network-workplace model? That is what I am looking at to work with along with a generalization of SEIR to perhaps account for assumptions cases. I would like to use NeuralODE to help fit the data. Seems like a good problem,

---

<div class="post-metadata">

**Author:** ![erlebach1](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@erlebach1](https://discourse.julialang.org/u/erlebach1)\
**Post date:** [May 24, 2020, 10:42pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/10 "2020-05-24T22:42:37Z")

</div>

I read the tutorials. In the context of SIR, I see where the transition rates fill in, but I do not see where the statistics Al properties of the time jumps are set. If the intervals between time events vary according to an exponential distribution with mean \lambda, where is that specified when using the various stochastic components of DifferentialEquations.jl?

Thanks,

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 24, 2020, 11:18pm UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/11 "2020-05-24T23:18:33Z")

</div>

> [@erlebach1](#):
>
> Have you ever worked with the Household-Network-workplace model?

I do not know that model.

> [@erlebach1](#):
>
> I see where the transition rates fill in, but I do not see where the statistics Al properties of the time jumps are set.

I plan to show in good detail how to do this soon, but it’s not entirely straightforward.

---

<div class="post-metadata">

**Author:** ![erlebach1](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@erlebach1](https://discourse.julialang.org/u/erlebach1)\
**Post date:** [May 25, 2020, 12:01am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/12 "2020-05-25T00:01:05Z")

</div>

I look forward to that day, Chris. But if the user is not setting the value, you must be doing this in the code. In a SIR model doesn’t this parameter of the exponential distribution relate to the average infection length? So you explicitly allow setting the model contact rate, but not the infection rate, which could also,depend on time. Why was this decision made? What source code could I examine to develop some intuition? There is a lot I still do not understand.

This is a whole new world for me.

I found the following callback.jl function in DiffEqJump.jl .

> @inline function time\_to\_next\_jump(u,p,t,rate)  
> randexp()/rate(u,p,t)  
> #rand(Exponential(1/rate(u,p,t)))  
> end
> 
> ConstantRateJumpCallback(next\_jump,c::ConstantRateJump,end\_time) = ConstantRateJumpCallback(next\_jump,end\_time,c.rate,c.affect!,c.save\_positions)
> 
> DiscreteCallback(c::ConstantRateJumpCallback) = DiscreteCallback(c,c,c.save\_positions)
> 
> function DiscreteCallback(u,p,t,c::ConstantRateJump,end\_time)  
> next\_jump = time\_to\_next\_jump(u,p,t,c.rate)  
> DiscreteCallback(ConstantRateJumpCallback(next\_jump,c,end\_time))  
> end

Looking more carefully at DiffEqJump.jl, I noticed

> @inline function (p::ConstantRateJumpCallback)(integrator) # affect!  
> p.affect!(integrator)  
> p.next\_jump = integrator.t + time\_to\_next\_jump(integrator.t,integrator.u,p.rate)  
> if p.next\_jump \< p.end\_time  
> add\_tstop!(integrator,p.next\_jump)  
> end  
> end

I do not understand how the call to time\_to\_next\_jump() can only have three arguments. It definition has four with no default arguments.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 25, 2020, 3:05am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/13 "2020-05-25T03:05:34Z")

</div>

> [@erlebach1](#):
>
> But if the user is not setting the value, you must be doing this in the code. In a SIR model doesn’t this parameter of the exponential distribution relate to the average infection length? So you explicitly allow setting the model contact rate, but not the infection rate, which could also,depend on time. Why was this decision made? What source code could I examine to develop some intuition?

The parameters are all of the rates for each compound Poisson process. These can depend on time, but then you can no longer use Gillespie’s algorithm so you’d have to use a VaraibleRateJump or tau-leaping.

> [@erlebach1](#):
>
> I do not understand how the call to time\_to\_next\_jump() can only have three arguments. It definition has four with no default arguments.

The call seems to have 4 arguments. What version of the code are you looking at?[https://github.com/SciML/DiffEqJump.jl/blob/master/src/aggregators/direct.jl#L89](https://github.com/SciML/DiffEqJump.jl/blob/master/src/aggregators/direct.jl#L89)

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [May 25, 2020, 3:29am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/14 "2020-05-25T03:29:31Z")

</div>

I do not know what verion I am looking up, but using the link you provide, I doubled back to callbacks.jl:  
[https://github.com/SciML/DiffEqJump.jl/blob/master/src/callbacks.jl](https://github.com/SciML/DiffEqJump.jl/blob/master/src/callbacks.jl)

Lines 15: 3 arguments.

p.next\_jump = integrator.t + time\_to\_next\_jump(integrator.t,integrator.u,p.rate)

Line 31: 3 arguments

| | next\_jump = time\_to\_next\_jump(u,p,t,c.rate) |
| --- | --- |
| | |

Lines 21: 4 arguments

# @inline function time\_to\_next\_jump(u,p,t,rate)

Regarding VariableRateJump: The documentation implied that this is what you use when the rate depends on the solution variables, implying that the rate would vary between two Poisson time points since the solutions change between two Poisson time points. Is this true? Please excuse the improper use of words since I am only learning this.  
:

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 25, 2020, 3:49am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/15 "2020-05-25T03:49:51Z")

</div>

That part isn’t being used because jumps have to be aggregated. @isaacsas I think we can just delete that portion?

> [@erlebach](#):
>
> Regarding VariableRateJump: The documentation implied that this is what you use when the rate depends on the solution variables, implying that the rate would vary between two Poisson time points since the solutions change between two Poisson time points. Is this true? Please excuse the improper use of words since I am only learning this.

Yes. The other jumps make the assumption that the rates are constant between events, and so you can use an exponential to get the next event time. VariableRateJumps drop that assumption, at a cost.

---

<div class="post-metadata">

**Author:** ![erlebach1](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@erlebach1](https://discourse.julialang.org/u/erlebach1)\
**Post date:** [May 25, 2020, 4:03am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/16 "2020-05-25T04:03:47Z")

</div>

Thanks. It is making more sense now. So the SDE library is a discrete process with fixed time steps?

Where can I read about the aggregation process?

I will stop for now :). I appreciate your patience.

Gordon

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 25, 2020, 4:28am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/17 "2020-05-25T04:28:31Z")

</div>

> [@erlebach1](#):
>
> Thanks. It is making more sense now. So the SDE library is a discrete process with fixed time steps?

It’s continuous and discrete (and mixed) with adaptive and fixed time steps.

> [@erlebach1](#):
>
> Where can I read about the aggregation process?

The basic one is the Gillespie direct method. We really need to do a refresher on our tutorials here, because [https://docs.sciml.ai/latest/tutorials/discrete\_stochastic\_example/](https://docs.sciml.ai/latest/tutorials/discrete_stochastic_example/) only scratches the surface, and all of [https://docs.sciml.ai/latest/types/jump\_types/#Constant-Rate-Jump-Aggregators-1](https://docs.sciml.ai/latest/types/jump_types/#Constant-Rate-Jump-Aggregators-1) are different SSAs.

This is a good overview of `Direct`: [http://users.abo.fi/ipetre/advcompmod/Lecture\_6.pdf](http://users.abo.fi/ipetre/advcompmod/Lecture_6.pdf)

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [May 25, 2020, 6:04am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/18 "2020-05-25T06:04:08Z")

</div>

I would like to mention [PiecewiseDeterministicMarkovProcesses.jl](https://github.com/rveltz/PiecewiseDeterministicMarkovProcesses.jl) as well

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 25, 2020, 6:10am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/19 "2020-05-25T06:10:22Z")

</div>

Indeed, that’s a good alternative to VariableRateJumps

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 25, 2020, 7:47am UTC](https://discourse.julialang.org/t/continous-markov-processes/40090/20 "2020-05-25T07:47:21Z")

</div>

> [@erlebach](#):
>
> Could somebody, possibly Tom, provide me with one or more examples of how to use the package (without all the @test macros, which confuse me.

If that is really me (that’s the most creative way I have seen my name mangled yet): the package was used for some exploratory work for research, and is currently dormant. It wasn’t a full suite for CTM processes anyway, just a subset of the relevant functionality.

[Next page](https://discourse.julialang.org/t/continous-markov-processes/40090.md?page=2)