# Checkpointing with Julia

**URL:** <https://discourse.julialang.org/t/checkpointing-with-julia/24026>\
**Category:** Julia at Scale\
**Created:** [May 9, 2019, 12:32pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026 "2019-05-09T12:32:29Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [May 9, 2019, 12:32pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/1 "2019-05-09T12:32:29Z")

</div>

The post today by @Xing_Shi_Cai leads me to ask - has anyone worked with checkpointing in Julia?  
A bit of Googling would suggest DMTCP would work [http://dmtcp.sourceforge.net/](http://dmtcp.sourceforge.net/)

I guess the real answer is - why don’t I give it a try instead of asking?

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [May 9, 2019, 3:50pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/2 "2019-05-09T15:50:23Z")

</div>

Can you give a brief description on the kind of checkpointing you’re referring to, and maybe a small example of a situation in which it’s beneficial? The Sourceforge page assumes prior knowledge of what checkpointing is.

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [May 9, 2019, 3:56pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/3 "2019-05-09T15:56:11Z")

</div>

[Nextjournal offers similar checkpoints](https://nextjournal.com/help/save-state) as an experimental feature via [criu](https://www.criu.org/Main_Page)🙂

If you happen to try it out, please let us know how it works for you and if you run into any bugs 😉

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [May 9, 2019, 4:24pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/4 "2019-05-09T16:24:37Z")

</div>

Sure. When running on an HPC cluster you run a job on many compute nodes. Julia makes this easy. Quite often compute jobs will run for days or ever weeks. If a compute node crashes during that time then all your work is lost.  
Applications will often write out an intermediate file following every N time steps.  
So you can use this file to restart your computation for the point before it failed - you do not have to start from the beginning.  
If you do not have an intermediate save file a checkpoint means that the state of the application on each compute node is written to disk. You then can start the application at that point.

With exascale compute cluster being almost ready to go, the probabilities of a node failure become very high. So there is huge interest in writing out these checkpoints quickly to ‘burst buffer’ filesystems.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 9, 2019, 4:31pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/5 "2019-05-09T16:31:39Z")

</div>

Oh interesting. I never even thought to look for something like this. When I needed something like this, I did it the old fashioned way once where I just wrote out several large txt files every so often with the critical information.

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [May 9, 2019, 10:47pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/6 "2019-05-09T22:47:18Z")

</div>

Nice description! I’m also wondering if there’s any way to auto-recover so the failed parts are re-started in another healthy node.

---

<div class="post-metadata">

**Author:** ![tbenst](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbenst/32/3029_2.png) [@tbenst](https://discourse.julialang.org/u/tbenst)\
**Post date:** [February 11, 2021, 6:45pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/7 "2021-02-11T18:45:32Z")

</div>

I’m curious if state of this has evolved in past two years? I found [https://github.com/hildebrandmw/Checkpoints.jl](https://github.com/hildebrandmw/Checkpoints.jl) which looks promising, although not maintained it would seem.

Edit: a few other resources:

- linux / container userspace approach: [CRIU - Checkpoint/Restore in user space - Red Hat Customer Portal](https://access.redhat.com/articles/2455211)
- python approach (not widely adopted): [https://github.com/a-rahimi/python-checkpointing2](https://github.com/a-rahimi/python-checkpointing2)

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [February 11, 2021, 10:14pm UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/8 "2021-02-11T22:14:42Z")

</div>

[Dagger.jl](https://github.com/JuliaParallel/Dagger.jl/) recently gained checkpoint/restore support, which lets you specify how and where to save data for each “thunk” (unit of work) in your computation. Of course, Dagger also has decent fault tolerance for when workers spontaneously die, and is designed for distributed and heterogeneous computing.

One day in the far future, CRIU support would be very cool to have.

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 15, 2021, 7:50am UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026/9 "2021-02-15T07:50:59Z")

</div>

> [@tbenst](#):
>
> curious if state of this has evolved in past two years

Now that basic Erlang/OTP like error-handling is implemented in [Actors.jl](https://github.com/JuliaActors/Actors.jl) (see [issue #16](https://github.com/JuliaActors/Actors.jl/issues/16) there and [description in the manual](https://juliaactors.github.io/Actors.jl/dev/errors/)), I have an issue about checkpointing and want to develop it in the next weeks:

> <https://github.com/JuliaActors/Actors.jl/issues/21>
>
> Now with basic error handling (see issue #16 and \[description in the manual\](htt…ps://juliaactors.github.io/Actors.jl/dev/errors/)) there is still an issue of maintaining/saving and restoring actor state at termination and restart. 
> 
> For actor and task restart (by supervisors) \`checkpoint\` and \`restore\` is an important option. Thus actor state can be restored at restart. 
> 
> \## Actor initialization and termination with user defined callback functions
> 
> \- \[x\] develop \`init!\` functionality,
> \- \[x\] develop \`term!\` functionality,
> \- \[x\] implement the restart strategy described below.
> 
> \## User-defined checkpointing:
> 
> \- \[x\] basic \`checkpointing\` actor,
> \- \[x\] \`checkpoint\` call,
> \- \[x\] \`restore\` call,
> \- \[x\] checkpointing hierarchy,
> \- \[x\] checkpointing interval for 2nd level.
> 
> \## Integration
> 
> \- \[\] tests,
> \- \[\] documentation,
> \- \[\] examples

I would appreciate some comments or any help on that. Thank you in advance!