# Using measure in MLJ to evaluate binary classifier

**URL:** <https://discourse.julialang.org/t/using-measure-in-mlj-to-evaluate-binary-classifier/67025>\
**Category:** New to Julia\
**Tags:** machine-learning, mlj\
**Created:** [August 26, 2021, 9:50am UTC](https://discourse.julialang.org/t/using-measure-in-mlj-to-evaluate-binary-classifier/67025 "2021-08-26T09:50:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Salvatore\_Cosentino](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/salvatore_cosentino/32/15418_2.png) [@Salvatore\_Cosentino](https://discourse.julialang.org/u/Salvatore_Cosentino)\
**Post date:** [August 26, 2021, 9:50am UTC](https://discourse.julialang.org/t/using-measure-in-mlj-to-evaluate-binary-classifier/67025/1 "2021-08-26T09:50:37Z")

</div>

Hello everybody,  
I am trying to use the `measures` methods in `MLJ` to evaluate binary classifications

I basically have integer vectors of 0 and 1 for ground truth (`gt` from now on) and predictions (`pred` from now on).  
When I try to use `confusion_matrix` using the integer vectors I get an error similar to the one described in this [post](https://discourse.julialang.org/t/mlj-confusion-matrix-methoderror/46856).

The problem is almost solved when I provide vectors of Strings use `categorical` to convert them.  
Following is a toy example:

```julia
using MLJ

predStr = ["fast", "fast", "slow"];
gtStr = ["slow", "fast", "slow"];

cmtx = confusion_matrix(categorical(gtStr), categorical(predStr))
┌ Warning: The classes are un-ordered,
│ using: negative='fast' and positive='slow'.
│ To suppress this warning, consider coercing to OrderedFactor.
└ @ MLJBase ~/.julia/packages/MLJBase/7hkEm/src/measures/confusion_matrix.jl:96
              ┌───────────────────────────┐
              │ Ground Truth │
┌─────────────┼─────────────┬─────────────┤
│ Predicted │ fast │ slow │
├─────────────┼─────────────┼─────────────┤
│ fast │ 1 │ 0 │
├─────────────┼─────────────┼─────────────┤
│ slow │ 1 │ 1 │
└─────────────┴─────────────┴─────────────┘

```

For the specific case above I would like to know order the classes and suppress the warning.

In general I would like to know there is a general way to provide input to the methods in `measures` in the `MLJ` package.  
So far, using the `categorical` function seem to work but would love to hear your opinion.

Thanks a lot for being such a great community!

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [August 26, 2021, 4:33pm UTC](https://discourse.julialang.org/t/using-measure-in-mlj-to-evaluate-binary-classifier/67025/2 "2021-08-26T16:33:39Z")

</div>

You can make your categorical arrays ordered:

```julia
using MLJ

predStr = categorical(
    ["fast", "fast", "slow"];
    ordered = true,
    levels = ["slow", "fast"]
)

gtStr = categorical(
    ["slow", "fast", "slow"];
    ordered = true,
    levels = ["slow", "fast"]
)

```

```julia
julia> confusion_matrix(predStr, gtStr)
              ┌───────────────────────────┐
              │ Ground Truth │
┌─────────────┼─────────────┬─────────────┤
│ Predicted │ slow │ fast │
├─────────────┼─────────────┼─────────────┤
│ slow │ 1 │ 0 │
├─────────────┼─────────────┼─────────────┤
│ fast │ 1 │ 1 │
└─────────────┴─────────────┴─────────────┘

```

Note that according to the [docstring for ConfusionMatrix](https://alan-turing-institute.github.io/MLJ.jl/dev/performance_measures/#MLJBase.ConfusionMatrix), the first argument should be the predicted values and the second argument should be the true values.

Let’s compare the scientific types of an ordered and an unordered categorical array:

```julia
julia> scitype(predStr)
AbstractVector{OrderedFactor{2}}

julia> scitype(categorical(["a", "b"]))
AbstractVector{Multiclass{2}}

```

Also note that you can make categorical arrays with integers:

```julia
pred = categorical([1, 1, 0]; ordered=true)
gt = categorical([0, 1, 0]; ordered=true)

```

```julia
julia> confusion_matrix(pred, gt)
              ┌───────────────────────────┐
              │ Ground Truth │
┌─────────────┼─────────────┬─────────────┤
│ Predicted │ 0 │ 1 │
├─────────────┼─────────────┼─────────────┤
│ 0 │ 1 │ 0 │
├─────────────┼─────────────┼─────────────┤
│ 1 │ 1 │ 1 │
└─────────────┴─────────────┴─────────────┘

```

---

<div class="post-metadata">

**Author:** ![Salvatore\_Cosentino](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/salvatore_cosentino/32/15418_2.png) [@Salvatore\_Cosentino](https://discourse.julialang.org/u/Salvatore_Cosentino)\
**Post date:** [August 31, 2021, 1:13am UTC](https://discourse.julialang.org/t/using-measure-in-mlj-to-evaluate-binary-classifier/67025/3 "2021-08-31T01:13:57Z")

</div>

Thanks a lot @CameronBieganek  
I should have read better the documentation regarding `scitype`

Thank again!