# Nested cross-validation

**URL:** <https://discourse.julialang.org/t/nested-cross-validation/65089>\
**Category:** Statistics\
**Created:** [July 22, 2021, 8:30am UTC](https://discourse.julialang.org/t/nested-cross-validation/65089 "2021-07-22T08:30:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [July 22, 2021, 8:30am UTC](https://discourse.julialang.org/t/nested-cross-validation/65089/1 "2021-07-22T08:30:43Z")

</div>

For my PhD, I’m mainly dealing with data containing 10 to 20 features and sample sizes of about 100 to 500. According to quite some literature, cross-validation (CV) is biased and should be replaced by nested cross-validation whenever you can, computationally, afford it (Krstajic et al, [2014](https://doi.org/10.1186/1758-2946-6-10); Vabalas et al, [2019](https://doi.org/10.1371/journal.pone.0224365)). I was just thinking about writing a paper where I manually compare 4 models. Something like comparing a linear model to two Turing.jl models and maybe a random forest.

So, as a sanity check: should I put the 4 models in a nested cross-validation loop to get an automated answer to the following questions.

1. Which models performs the best?
2. How good will the best model perform?

I expect that runtime will be okay. Compared to cross-validation, runtime is only multiplied by the number of models than I want to compare and the outer loop can run in parallel.

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [July 23, 2021, 4:01pm UTC](https://discourse.julialang.org/t/nested-cross-validation/65089/2 "2021-07-23T16:01:00Z")

</div>

Hi Rikh, a few comments if I may,

> [@rikh](#):
>
> cross-validation (CV) is biased and should be replaced by nested cross-validation whenever you can

Perhaps just whenever you use it to select a model? Just for validating a model CV should be fine I believe.

> [@rikh](#):
>
> So, as a sanity check: should I put the 4 models in a nested cross-validation loop to get an automated answer to the following questions.

Be careful with automated answers unless you’re okay with automated mistakes.

> [@rikh](#):
>
> Which models performs the best?

I would say this question makes proper sense when stated as “Which models perform best **in this very specific data set**?”. You’ll find that any of those four models you’re considering will perform best or not based on which data set you choose. This is specially true if your paper is related to QSAR data sets like Krstajic et al, [2014].

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [July 23, 2021, 8:23pm UTC](https://discourse.julialang.org/t/nested-cross-validation/65089/3 "2021-07-23T20:23:26Z")

</div>

> [@viraltux](#):
>
> Perhaps just whenever you use it to select a model? Just for validating a model CV should be fine I believe.

Hmm, I just thought it was the other way around. But, I’m also not sure.

Very good comment about the automated mistakes!

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [October 13, 2021, 9:45am UTC](https://discourse.julialang.org/t/nested-cross-validation/65089/4 "2021-10-13T09:45:21Z")

</div>

Nested cross-validation was added to MLJ a while ago thanks to [Anthony Blaom](https://github.com/ablaom).

It can be done via a `TunedModel` ([Tuning Models · MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/tuning_models/#explicit)). From the docs, to run 2 outer folds and 3 inner folds:

```julia
tree = (@load DecisionTreeClassifier pkg=DecisionTree verbosity=0)()
knn = (@load KNNClassifier pkg=NearestNeighborModels verbosity=0)()
models = [tree, knn]

# For each model, run 3-fold cross-validation (inner loop).
multi_model = TunedModel(models=models,
						 resampling=CV(nfolds=3),
						 measure=log_loss,
						 check_measure=false)

# For each trail, run 2-fold cross validation (outer loop).
e = evaluate(multi_model, X, y,
			 resampling=CV(nfolds=2),
			 measure=log_loss,
			 verbosity=6)

# Best model for the first trail.
e.report_per_fold[1].best_model

```

For more information, see also the example in [scikit-learn](https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html).