# MLJ: Evaluating a probabilistic metric and a deterministic metric at the same time
**URL:** https://discourse.julialang.org/t/mlj-evaluating-a-probabilistic-metric-and-a-deterministic-metric-at-the-same-time/39775
**Category:** Machine Learning
**Tags:** mlj
**Created:** [May 19, 2020, 7:23pm UTC](https://discourse.julialang.org/t/mlj-evaluating-a-probabilistic-metric-and-a-deterministic-metric-at-the-same-time/39775 "2020-05-19T19:23:24Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [May 19, 2020, 7:23pm UTC](https://discourse.julialang.org/t/mlj-evaluating-a-probabilistic-metric-and-a-deterministic-metric-at-the-same-time/39775/1 "2020-05-19T19:23:25Z")
The `evaluate` and `evaluate!` methods in MLJ can accept a vector of metric functions. However, it doesn’t appear that you can evaluate metrics based on probabilistic predictions (e.g. AUC) and metrics based on deterministic predictions (e.g. accuracy) at the same time. Here’s a MWE:
```julia
using DataFrames
using RDatasets
using MLJ
using MLJLinearModels
iris = dataset("datasets", "iris")
df = filter(r -> r.Species != "virginica", iris)
y = droplevels!(copy(df.Species))
X = select(df, Not(:Species))
model = LogisticClassifier(penalty=:none)
logistic_machine = machine(model, X, y)
holdout = Holdout(shuffle=true, rng=1)
logistic_auc = evaluate!(
logistic_machine,
resampling = holdout,
measure = auc
)
logistic_accuracy = evaluate!(
logistic_machine,
resampling = holdout,
operation = predict_mode,
measure = accuracy
)
```
Does anyone know if it’s possible to evaluate `auc` and `accuracy` at the same time without having to run `evaluate!` twice?
---
### Author:  [@samuel\_okon](https://discourse.julialang.org/u/samuel_okon)
#### Post date: [May 20, 2020, 11:01am UTC](https://discourse.julialang.org/t/mlj-evaluating-a-probabilistic-metric-and-a-deterministic-metric-at-the-same-time/39775/2 "2020-05-20T11:01:07Z")
In MLJ `accuracy` measure is only defined for deterministic classifiers. You could define your custom accuracy measure that works on probabilistic classifiers using the code below.
```julia
custom_accuracy(yhat, y) = accuracy(mode.(yhat), y)
MLJ.reports_each_observation(::typeof(custom_accuracy)) = false
MLJ.supports_weights(::typeof(custom_accuracy)) = true
MLJ.orientation(::typeof(custom_accuracy)) = :score
MLJ.is_feature_dependent(::typeof(custom_accuracy)) = :false
MLJ.prediction_type(::typeof(custom_accuracy)) = :probabilistic
```
Then you could then do
```julia
logistic_auc_accuracy = evaluate!(
logistic_machine,
resampling = holdout,
measure = [auc, custom_accuracy]
)
```
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [May 20, 2020, 8:51pm UTC](https://discourse.julialang.org/t/mlj-evaluating-a-probabilistic-metric-and-a-deterministic-metric-at-the-same-time/39775/3 "2020-05-20T20:51:43Z")
Awesome, thanks @samuel_okon! That’s a good solution. Though since MLJ measures have a `prediction_type` trait, it seems like it might be possible to extend `evaluate` to accept measures for different prediction types and have `evaluate` automatically run each of the necessary types of prediction. Maybe I’ll make a PR for that. 🙂
---
### Author:  [@samuel\_okon](https://discourse.julialang.org/u/samuel_okon)
#### Post date: [May 20, 2020, 9:02pm UTC](https://discourse.julialang.org/t/mlj-evaluating-a-probabilistic-metric-and-a-deterministic-metric-at-the-same-time/39775/4 "2020-05-20T21:02:19Z")
That sounds nice. But the problem is that these measures were defined for either `Deterministic` or `Probabilistic` classifiers not both. Also applying a `Probabilistic` measure on `Deterministic` outputs won’t be well defined since vector of `UnivariateFinite` is needed.
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [May 20, 2020, 9:45pm UTC](https://discourse.julialang.org/t/mlj-evaluating-a-probabilistic-metric-and-a-deterministic-metric-at-the-same-time/39775/5 "2020-05-20T21:45:00Z")
Yeah, I was thinking that if `measure = [auc, accuracy]`, then maybe internally `evaluate` could run both `predict` and `predict_mode` to get the two separate types of prediction. Then it could use the proper type of prediction for each metric.