# MLJ: how to read/interpret \`measure\` from evaluate
**URL:** https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689
**Category:** General Usage
**Tags:** question, mlj
**Created:** [November 3, 2022, 3:53am UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689 "2022-11-03T03:53:29Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 3, 2022, 3:53am UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/1 "2022-11-03T03:53:29Z")
In this toy example I know one config is overtraining:
```julia
tree1 = EvoTreeClassifier(; nrounds = 10, eta = 0.1, max_depth=5)
tree2 = EvoTreeClassifier(; nrounds = 10, eta = 0.1, max_depth=10)
```
because I checked it via manual training and fitting:
```julia
mach = machine(tree, X, y)
train, test = partition(eachindex(y), 0.7);
fit!(mach, rows=train);
y_pred = predict(mach, @view X[test, :]);
fprs, tprs, ts = roc_curve(y_pred, @view y[test]);
f = lines(fprs, tprs; label="test")
y_pred = predict(mach, @view X[train, :]);
fprs, tprs, ts = roc_curve(y_pred, @view y[train]);
lines!(fprs, tprs; label="train")
axislegend()
```
Expand to see plots:
> **Two Plots**
>
> 
> 
But I can’t tell from looking at result from `evaluate(tree)`:
```julia
evaluate(tree1, X, y,
resampling=CV(; nfolds=6, shuffle=nothing),
measure=[log_loss, auc],
verbosity=1)
┌────────────────────────────────┬───────────┬─────────────┬─────────┬──────────
│ measure │ operation │ measurement │ 1.96*SE │ per_fol ⋯
├────────────────────────────────┼───────────┼─────────────┼─────────┼──────────
│ LogLoss( │ predict │ 0.509 │ 0.00369 │ [0.516, ⋯
│ tol = 2.220446049250313e-16) │ │ │ │ ⋯
│ AreaUnderCurve() │ predict │ 0.831 │ 0.00429 │ [0.823, ⋯
└────────────────────────────────┴───────────┴─────────────┴─────────┴──────────
#res.per_fold
2-element Vector{Vector}:
[0.5164007967507921, 0.503903753403266, 0.5087914373970631, 0.5097046862399631, 0.5067821829301071, 0.5107959181599303]
Any[0.8225134347902088, 0.8373614101583923, 0.8336094740471997, 0.8313957836206489, 0.831747928950465, 0.8315899300438203]
evaluate(tree2, X, y,
resampling=CV(; nfolds=6, shuffle=nothing),
measure=[log_loss, auc],
verbosity=1)
┌────────────────────────────────┬───────────┬─────────────┬─────────┬──────────
│ measure │ operation │ measurement │ 1.96*SE │ per_fol ⋯
├────────────────────────────────┼───────────┼─────────────┼─────────┼──────────
│ LogLoss( │ predict │ 0.484 │ 0.0037 │ [0.489, ⋯
│ tol = 2.220446049250313e-16) │ │ │ │ ⋯
│ AreaUnderCurve() │ predict │ 0.851 │ 0.00314 │ [0.845, ⋯
└────────────────────────────────┴───────────┴─────────────┴─────────┴──────────
[0.48938241998998006, 0.4799705129014878, 0.4845229947659094, 0.48114972544923074, 0.4808197206490011, 0.48903871416415157]
Any[0.8452396760282181, 0.8533083034135965, 0.8519176930326134, 0.8538765775152372, 0.8520602768012421, 0.846907808652614]
```
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 3, 2022, 3:55am UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/2 "2022-11-03T03:55:17Z")
I don’t know if my eyes are not trained enough, but I can’t decide what’s a “good” value from a “bad” value which I thought should be very simple given the ROC curve difference
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 3, 2022, 4:54am UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/3 "2022-11-03T04:54:09Z")
same underlying mechanism causing problems for model hyperparameter tunning:
```julia
tree = EvoTreeClassifier()
rs = [
range(tree, :eta, lower=0.001, origin=0.002, upper=0.1),
range(tree, :nrounds, lower=10, origin=20, upper=250),
range(tree, :min_weight, lower=0.5, origin=1.0, upper=3.5),
range(tree, :colsample, lower=0.3, origin=0.8, upper=1.0),
range(tree, :rowsample, lower=0.3, origin=0.8, upper=1.0),
range(tree, :max_depth, lower=3, origin=4, upper=8),
];
tm = TunedModel(;
model=tree,
resampling = CV(nfolds=3),
tuning=Grid(goal=15),
range=rs, measure=[auc, log_loss]
);
tunemach = machine(tm, X, y);
>fit!(tunemach; rows=train)
┌ Info: Training machine(ProbabilisticTunedModel(model = EvoTreeClassifier(nrounds = 10, …), …), …).
└ @ MLJBase /home/jiling/.julia/packages/MLJBase/6ooqv/src/machines.jl:496
┌ Info: Attempting to evaluate 64 models.
└ @ MLJTuning /home/jiling/.julia/packages/MLJTuning/eSEHt/src/tuned_models.jl:727
Evaluating over 64 metamodels: 100%[=========================] Time: 0:05:12
>fitted_params(tunemach).best_model
EvoTreeClassifier(
nrounds = 250,
lambda = 0.0,
gamma = 0.0,
eta = 0.1,
max_depth = 8,
min_weight = 3.5,
rowsample = 1.0,
colsample = 1.0,
nbins = 32,
alpha = 0.5,
rng = Random.MersenneTwister(123, (0, 23413734, 23412732, 18)),
device = "cpu")
```
---
### Author:  [@samuel\_okon](https://discourse.julialang.org/u/samuel_okon)
#### Post date: [November 3, 2022, 4:31pm UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/4 "2022-11-03T16:31:14Z")
Checking for overfitting this way can be misleading.
1. For model selection depending on the application domain, it might be less important to check for closeness of the training and test performance see [here](https://machinelearningmastery.com/overfitting-machine-learning-models). It these cases it is sufficient to compare the out-of-sample scores of the two models. It seems plot 2 has a higher test AUC than plot 1 (I may be wrong. Try to plot both on the same pane).
2. It might just that the plots above were due the nature of the actual train and test datasets used.
I would like to see the result of:
```julia
evaluate(tree1, X[train, :], y,
resampling=CV(; nfolds=6, shuffle=nothing),
measure=[log_loss, auc],
verbosity=1)
evaluate(tree2, X[train, :], y,
resampling=CV(; nfolds=6, shuffle=nothing),
measure=[log_loss, auc],
verbosity=1)
```
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 3, 2022, 5:29pm UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/5 "2022-11-03T17:29:15Z")
here’s the result:
```julia
evaluate(tree1, X[train, :], y[train],
resampling=CV(; nfolds=6, shuffle=nothing),
measure=[log_loss, auc],
verbosity=1)
┌────────────────────────────────┬───────────┬─────────────┬─────────┬──────────
│ measure │ operation │ measurement │ 1.96*SE │ per_fol ⋯
├────────────────────────────────┼───────────┼─────────────┼─────────┼──────────
│ LogLoss( │ predict │ 0.511 │ 0.00473 │ [0.516, ⋯
│ tol = 2.220446049250313e-16) │ │ │ │ ⋯
│ AreaUnderCurve() │ predict │ 0.831 │ 0.00608 │ [0.823, ⋯
└────────────────────────────────┴───────────┴─────────────┴─────────┴──────────
res.per_fold
2-element Vector{Vector}:
[0.5156692322207046, 0.5148959990683522, 0.5016058812139339, 0.5142813858720531, 0.508294783136379, 0.5092500658411819]
Any[0.8230607740610442, 0.8246094193656129, 0.8414543292511101, 0.8274620221394415, 0.8349777148818776, 0.8316299858331504]
#########################################
evaluate(tree2, X[train, :], y[train],
resampling=CV(; nfolds=6, shuffle=nothing),
measure=[log_loss, auc],
verbosity=1)
┌────────────────────────────────┬───────────┬─────────────┬─────────┬──────────
│ measure │ operation │ measurement │ 1.96*SE │ per_fol ⋯
├────────────────────────────────┼───────────┼─────────────┼─────────┼──────────
│ LogLoss( │ predict │ 0.49 │ 0.00448 │ [0.493, ⋯
│ tol = 2.220446049250313e-16) │ │ │ │ ⋯
│ AreaUnderCurve() │ predict │ 0.844 │ 0.00445 │ [0.84, ⋯
└────────────────────────────────┴───────────┴─────────────┴─────────┴──────────
res.per_fold
[0.49313997400549425, 0.4960096759426803, 0.48619524416447407, 0.49555493107760795, 0.4851382554812855, 0.48597214007449996]
Any[0.8401277524445331, 0.8384721512647232, 0.8486174757494099, 0.8400657842593326, 0.8494939978226504, 0.8480602057851294]
```
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 3, 2022, 5:32pm UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/6 "2022-11-03T17:32:46Z")
> [@samuel\_okon](#):
>
> For model selection depending on the application domain, it might be less important to check for closeness of the training and test performance see [here](https://machinelearningmastery.com/overfitting-machine-learning-models).
this is not gonna work, the training set will be part of our final evaluation set.
Unless, I do cross training and evaluate data only in the model that hasn’t seen it? is that legal?
---
### Author:  [@samuel\_okon](https://discourse.julialang.org/u/samuel_okon)
#### Post date: [November 3, 2022, 7:14pm UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/7 "2022-11-03T19:14:15Z")
What I’m suggesting is doing cross validation on the training dataset. The test dataset isn’t used here.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 3, 2022, 7:21pm UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/8 "2022-11-03T19:21:59Z")
in the end the training dataset and test set both must be used.
say, I only have 10k data points and I have to use all 10k in the end, so if I CV on 9k and test on 1k, the gap between 9k and 1k still matters because in the end 90% of the data will have fake high performance
---
### Author:  [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)
#### Post date: [November 4, 2022, 12:27am UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/9 "2022-11-04T00:27:38Z")
The reinforce what I think @samuel_okon is suggesting, a typical setup would involve considering 3 types of data: train, eval and test.
Let’s assume that out of this 10k, 2k are to be used as test data. Then, on the remaining 8k oservations, you could perform 4-fold CV. Each of the 8K observations would be assigned to one of the 4 CV bucket. Then for each of the 4 CV, a model is fitted on 6K data until it starts overfittnig on the 2k eval data. The selected model would be the one which consistently produced best performance throughout the 4 CVs. Then final assessment would be done on the 2k test that was initially set aside by training the selected model on the full 8K training data.
It’s still possible to assess performance on the training data by looking at the metric of each CV model on their eval data. However, this is exposed to some bias as typically, the early stopping cireteria comes from the performance on that eval data, hence there’s some information leakage.
---
### Author:  [@jbrea](https://discourse.julialang.org/u/jbrea)
#### Post date: [November 4, 2022, 9:21am UTC](https://discourse.julialang.org/t/mlj-how-to-read-interpret-measure-from-evaluate/89689/10 "2022-11-04T09:21:57Z")
If you _only_ want to find the best hyperparameters, you don’t need to split into training and test data, but you can use all the data to do cross-validation. If you want to find the best hyperparameters _and_ compute an estimate of the test error, you can use e.g. nested cross-validation.