For my PhD, I’m mainly dealing with data containing 10 to 20 features and sample sizes of about 100 to 500. According to quite some literature, cross-validation (CV) is biased and should be replaced by nested cross-validation whenever you can, computationally, afford it (Krstajic et al, 2014; Vabalas et al, 2019). I was just thinking about writing a paper where I manually compare 4 models. Something like comparing a linear model to two Turing.jl models and maybe a random forest.
So, as a sanity check: should I put the 4 models in a nested cross-validation loop to get an automated answer to the following questions.
- Which models performs the best?
- How good will the best model perform?
I expect that runtime will be okay. Compared to cross-validation, runtime is only multiplied by the number of models than I want to compare and the outer loop can run in parallel.