# How do I regularise MLJFlux models?

**URL:** https://discourse.julialang.org/t/how-do-i-regularise-mljflux-models/82349
**Category:** Machine Learning
**Tags:** mlj
**Created:** [June 6, 2022, 11:14pm UTC](https://discourse.julialang.org/t/how-do-i-regularise-mljflux-models/82349 "2022-06-06T23:14:12Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)
#### Post date: [June 6, 2022, 11:14pm UTC](https://discourse.julialang.org/t/how-do-i-regularise-mljflux-models/82349/1 "2022-06-06T23:14:12Z")

</div>

Reposting this question from a Slack channel.

---

<div class="post-metadata">

### Author: ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)
#### Post date: [June 6, 2022, 11:18pm UTC](https://discourse.julialang.org/t/how-do-i-regularise-mljflux-models/82349/2 "2022-06-06T23:18:00Z")

</div>

The question of regularization for neural networks is a bit  
complicated, and I’m no expert. It has frequently been observed that  
increasing the total number of weights (increasing complexity) does  
not necessarily lead to over-fitting, but the phenomenon is poorly  
understood in general. In [this paper](https://www.pnas.org/doi/10.1073/pnas.1903070116) Belkin et  
al. (2019) introduce specific examples of networks where a “double  
descent risk curve” should be expected (so no over-fitting). However,  
in [this preprint](https://arxiv.org/abs/2010.09610v1?utm_campaign=AI%20Scholar%20Weekly%20&utm_medium=email&utm_source=Revue%20newsletter) Nicahni et al. (2020) argue that while increasing network _width_ may not  
lead to over-fitting, increasing _depth_ can still lead to over-fitting.

Returning to the question, regularization options in MLJFlux/Flux are:

- Early stopping: You end training when an out-of-sample error begins  
to deteriorate. MLJ’s `IteratedModel` wrapper is useful for  
automating this. See, the Boston or MNIST examples  
[here](https://github.com/FluxML/MLJFlux.jl/tree/dev/examples).

- Add `Dropout` layers to your Flux model (aka chain) (through the  
`builder` hyper-parameter of your flux model). See [Normalization &  
Regularization](https://fluxml.ai/Flux.jl/stable/models/layers/#Normalisation-and-Regularisation)  
section of the Flux manual

- Add L1/L2 weight penalty regularization by specifying appropriate  
values of the hyper-parameters `lambda` (strength of regularization)  
and `alpha` of your MLJFlux model (L2/L1 mix). If `alpha=0` then there is only L2  
regularization