# \[ANN\] EarlyStopping 0.1.0

**URL:** <https://discourse.julialang.org/t/ann-earlystopping-0-1-0/55761>\
**Category:** Package Announcements\
**Tags:** machine-learning\
**Created:** [February 22, 2021, 4:26am UTC](https://discourse.julialang.org/t/ann-earlystopping-0-1-0/55761 "2021-02-22T04:26:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [February 22, 2021, 4:26am UTC](https://discourse.julialang.org/t/ann-earlystopping-0-1-0/55761/1 "2021-02-22T04:26:55Z")

</div>

The `EarlyStopper` objects defined in [EarlyStopping.jl](https://github.com/ablaom/EarlyStopping.jl) consume a sequence of numbers called _losses_ generated by some external algorithm - generally the training loss or out-of-sample loss of some iterative statistical model - and decide when those losses have dropped sufficiently to warrant terminating the algorithm.

The package is mainly intended for developers in the ML space. There is [a plan](https://github.com/alan-turing-institute/MLJ.jl/issues/139) to use it in MLJ.jl in a model wrapper for controlling iterative models (including “self-tuning” models).

A number of commonly applied _stopping criteria_ are included out-of-the-box, including all those surveyed in the paper [Prechelt, Lutz (1998)](https://link.springer.com/chapter/10.1007%2F3-540-49430-8_3):  
“Early Stopping - But When?”, in _Neural Networks: Tricks of the Trade_, ed. G. Orr, Springer.

| criterion | description | notation in Prechelt |
| --- | --- | --- |
| `Never()` | Never stop | |
| `NotANumber()` | Stop when `NaN` encountered | |
| `TimeLimit(t=0.5)` | Stop after `t` hours | |
| `GL(alpha=2.0)` | Stop after “Generalization Loss” exceeds `alpha` | `GL_α` |
| `PQ(alpha=0.75, k=5)` | Stop after “Progress-modified GL” exceeds `alpha` | `PQ_α` |
| `Patience(n=5)` | Stop after `n` consecutive loss increases | `UP_s` |
| `Disjunction(c...)` | Stop when any of the criteria `c` apply | |

---

<div class="post-metadata">

**Author:** ![dmolina](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmolina/32/5246_2.png) [@dmolina](https://discourse.julialang.org/u/dmolina)\
**Post date:** [February 23, 2021, 6:16pm UTC](https://discourse.julialang.org/t/ann-earlystopping-0-1-0/55761/2 "2021-02-23T18:16:03Z")

</div>

It is nice!

It is very useful to be able to stop after a certain time, of after several worse results. However, I miss a criterion for a maximum number, not only when the results got worse, in any case. Also, it is not clear if several criteria can be combined.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [February 23, 2021, 7:53pm UTC](https://discourse.julialang.org/t/ann-earlystopping-0-1-0/55761/3 "2021-02-23T19:53:21Z")

</div>

Thanks for the feedback!

Yes, can can combine multiple criteria. Just give the `EarlyStopper` constructor multiple arguments as in first readme example. You can call `message` on the stopper to see which one applied.

Happy to add new criteria - just open an issue on the repo. PR’s also welcome.

---

<div class="post-metadata">

**Author:** ![dmolina](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmolina/32/5246_2.png) [@dmolina](https://discourse.julialang.org/u/dmolina)\
**Post date:** [February 24, 2021, 7:05pm UTC](https://discourse.julialang.org/t/ann-earlystopping-0-1-0/55761/4 "2021-02-24T19:05:21Z")

</div>

You are right about the multiple criteria, I did not realise that you can combine all criteria in the same EarlyStopper.

Thank you for be open to my suggestion, I have just done a [PR implementing my suggestion](https://github.com/ablaom/EarlyStopping.jl/pull/3). Initially the name is MaximumChecks, but I am open to rename it. It is a nice package.