MLJ: Evaluating a probabilistic metric and a deterministic metric at the same time

Specific Domains Machine Learning
2

In MLJ accuracy measure is only defined for deterministic classifiers. You could define your custom accuracy measure that works on probabilistic classifiers using the code below.

custom_accuracy(yhat, y) = accuracy(mode.(yhat), y)
MLJ.reports_each_observation(::typeof(custom_accuracy)) = false
MLJ.supports_weights(::typeof(custom_accuracy)) = true
MLJ.orientation(::typeof(custom_accuracy)) = :score 
MLJ.is_feature_dependent(::typeof(custom_accuracy)) = :false
MLJ.prediction_type(::typeof(custom_accuracy)) = :probabilistic

Then you could then do

logistic_auc_accuracy = evaluate!(
    logistic_machine,
    resampling = holdout,
    measure = [auc, custom_accuracy]
)
5 Likes