In this toy example I know one config is overtraining:
tree1 = EvoTreeClassifier(; nrounds = 10, eta = 0.1, max_depth=5)
tree2 = EvoTreeClassifier(; nrounds = 10, eta = 0.1, max_depth=10)
because I checked it via manual training and fitting:
mach = machine(tree, X, y)
train, test = partition(eachindex(y), 0.7);
fit!(mach, rows=train);
y_pred = predict(mach, @view X[test, :]);
fprs, tprs, ts = roc_curve(y_pred, @view y[test]);
f = lines(fprs, tprs; label="test")
y_pred = predict(mach, @view X[train, :]);
fprs, tprs, ts = roc_curve(y_pred, @view y[train]);
lines!(fprs, tprs; label="train")
axislegend()
Expand to see plots:
But I canβt tell from looking at result from evaluate(tree)
:
evaluate(tree1, X, y,
resampling=CV(; nfolds=6, shuffle=nothing),
measure=[log_loss, auc],
verbosity=1)
ββββββββββββββββββββββββββββββββββ¬ββββββββββββ¬ββββββββββββββ¬ββββββββββ¬ββββββββββ
β measure β operation β measurement β 1.96*SE β per_fol β―
ββββββββββββββββββββββββββββββββββΌββββββββββββΌββββββββββββββΌββββββββββΌββββββββββ
β LogLoss( β predict β 0.509 β 0.00369 β [0.516, β―
β tol = 2.220446049250313e-16) β β β β β―
β AreaUnderCurve() β predict β 0.831 β 0.00429 β [0.823, β―
ββββββββββββββββββββββββββββββββββ΄ββββββββββββ΄ββββββββββββββ΄ββββββββββ΄ββββββββββ
#res.per_fold
2-element Vector{Vector}:
[0.5164007967507921, 0.503903753403266, 0.5087914373970631, 0.5097046862399631, 0.5067821829301071, 0.5107959181599303]
Any[0.8225134347902088, 0.8373614101583923, 0.8336094740471997, 0.8313957836206489, 0.831747928950465, 0.8315899300438203]
evaluate(tree2, X, y,
resampling=CV(; nfolds=6, shuffle=nothing),
measure=[log_loss, auc],
verbosity=1)
ββββββββββββββββββββββββββββββββββ¬ββββββββββββ¬ββββββββββββββ¬ββββββββββ¬ββββββββββ
β measure β operation β measurement β 1.96*SE β per_fol β―
ββββββββββββββββββββββββββββββββββΌββββββββββββΌββββββββββββββΌββββββββββΌββββββββββ
β LogLoss( β predict β 0.484 β 0.0037 β [0.489, β―
β tol = 2.220446049250313e-16) β β β β β―
β AreaUnderCurve() β predict β 0.851 β 0.00314 β [0.845, β―
ββββββββββββββββββββββββββββββββββ΄ββββββββββββ΄ββββββββββββββ΄ββββββββββ΄ββββββββββ
[0.48938241998998006, 0.4799705129014878, 0.4845229947659094, 0.48114972544923074, 0.4808197206490011, 0.48903871416415157]
Any[0.8452396760282181, 0.8533083034135965, 0.8519176930326134, 0.8538765775152372, 0.8520602768012421, 0.846907808652614]