# How to properly use MLJBase.roc\_curve

**URL:** <https://discourse.julialang.org/t/how-to-properly-use-mljbase-roc-curve/90648>\
**Category:** New to Julia\
**Tags:** question, package, mlj\
**Created:** [November 22, 2022, 3:02pm UTC](https://discourse.julialang.org/t/how-to-properly-use-mljbase-roc-curve/90648 "2022-11-22T15:02:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![adz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adz/32/33773_2.png) [@adz](https://discourse.julialang.org/u/adz)\
**Post date:** [November 22, 2022, 3:02pm UTC](https://discourse.julialang.org/t/how-to-properly-use-mljbase-roc-curve/90648/1 "2022-11-22T15:02:12Z")

</div>

Hi all, I’m getting used to ML and Julia and am wondering how to properly make use of MLJ’s api. Specifically, I’m not sure what exactly it wants as arguments to MLJ.roc\_curve. Here is some code I use to train and evaluate a model:

```julia

using DataFrames, CSV, MLJ, DecisionTree, MLDataUtils, Pipe

df = DataFrame(CSV.File("./data.csv"))
df = @pipe (DataFrames.transform(df, [:DGN, :PRE6] .=> x -> parse.(UInt32, SubString.(x, Ref(4))), renamecols=false) |>
    DataFrames.transform(_, :PRE14 => x -> parse.(UInt32, SubString.(x, Ref(3:4))), renamecols=false))

using MLDataUtils

y, X = unpack(df[!, Not("id")], ==(:Risk1Yr); rng=123);
(X_train, y_train), (X_val, y_val), (X_test, y_test) = stratifiedobs((X, y), p=(0.7, 0.2));

rf = RandomForestClassifier()
DecisionTree.fit!(rf, Matrix(X_train), y_train)

preds = DecisionTree.predict_proba(rf, Matrix(X_val))
MLJ.roc_curve(preds[:,2], y_val)

```

I get the following error:

```julia
MethodError: no method matching pdf(::Float64, ::Bool)
Closest candidates are:
  pdf(!Matched::Distributions.Logistic, ::Real) at ~/.julia/packages/Distributions/Vkexj/src/univariate/continuous/logistic.jl:81
  pdf(!Matched::KernelDensity.BivariateKDE, ::Any, !Matched::Any) at ~/.julia/packages/KernelDensity/bNBAQ/src/interp.jl:32
  pdf(!Matched::Distributions.Truncated, ::Real) at ~/.julia/packages/Distributions/Vkexj/src/truncate.jl:133
  ...

Stacktrace:
 [1] _broadcast_getindex_evalf
   @ ./broadcast.jl:670 [inlined]
 [2] _broadcast_getindex
   @ ./broadcast.jl:643 [inlined]
 [3] getindex
   @ ./broadcast.jl:597 [inlined]
 [4] copy
   @ ./broadcast.jl:899 [inlined]
 [5] materialize
   @ ./broadcast.jl:860 [inlined]
 [6] roc_curve(ŷm::Vector{Float64}, ym::SubArray{Bool, 1, Vector{Bool}, Tuple{Vector{Int64}}, false})
   @ MLJBase ~/.julia/packages/MLJBase/U4Dis/src/measures/roc.jl:48
 [7] top-level scope
   @ ~/code/julia/mltest/test_mlj.ipynb:17

```

Any advice?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [November 22, 2022, 3:47pm UTC](https://discourse.julialang.org/t/how-to-properly-use-mljbase-roc-curve/90648/2 "2022-11-22T15:47:11Z")

</div>

This has definitely bitten me before - `roc_curve` doesn’t have great documentation and the error is pretty unhelpful. It works if you stay in the MLJ ecosystem - I believe it expects predictions in the form of MLJ’s own `UnivariateFinite` type, i.e. a distribution over classes (although @ablaom will be able to confirm whether this is true).

Here’s a full MWE:

```julia
julia> using MLJ, MLJDecisionTreeInterface

julia> X = rand(100, 3); y = rand(Bool, 100);

julia> Tree = @load RandomForestClassifier pkg=DecisionTree
[ Info: For silent loading, specify `verbosity=0`.
import MLJDecisionTreeInterface ✔
RandomForestClassifier

julia> tree = Tree()
RandomForestClassifier(
  max_depth = -1,
  min_samples_leaf = 1,
  min_samples_split = 2,
  min_purity_increase = 0.0,
  n_subfeatures = -1,
  n_trees = 10,
  sampling_fraction = 0.7,
  feature_importance = :impurity,
  rng = Random._GLOBAL_RNG())

julia> mach = machine(tree, (x1 = X[:, 1], x2 = X[:, 2]), categorical(y))
untrained Machine; caches model-specific representations of data
  model: RandomForestClassifier(max_depth = -1, …)
  args:
    1: Source @514 ⏎ Table{AbstractVector{Continuous}}
    2: Source @659 ⏎ AbstractVector{Multiclass{2}}

julia> fit!(mach)
[ Info: Training machine(RandomForestClassifier(max_depth = -1, …), …).
trained Machine; caches model-specific representations of data
  model: RandomForestClassifier(max_depth = -1, …)
  args:
    1: Source @514 ⏎ Table{AbstractVector{Continuous}}
    2: Source @659 ⏎ AbstractVector{Multiclass{2}}

julia> ŷ = predict(mach);

julia> typeof(ŷ)
UnivariateFiniteVector{Multiclass{2}, Bool, UInt32, Float64}

julia> roc_curve(ŷ, y)
([0.0, 0.034482758620689655, 0.10344827586206896, 0.13793103448275862, 0.1724137931034483, 0.22413793103448276, 0.25862068965517243, 0.3620689655172414, 0.41379310344827586, 0.5, 1.0], [0.0, 0.47619047619047616, 0.5714285714285714, 0.6428571428571429, 0.6428571428571429, 0.7142857142857143, 0.7857142857142857, 0.8095238095238095, 0.8333333333333334, 0.8333333333333334, 1.0], [1.0, 0.9, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0])

```

---

<div class="post-metadata">

**Author:** ![adz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adz/32/33773_2.png) [@adz](https://discourse.julialang.org/u/adz)\
**Post date:** [November 22, 2022, 4:55pm UTC](https://discourse.julialang.org/t/how-to-properly-use-mljbase-roc-curve/90648/3 "2022-11-22T16:55:02Z")

</div>

Thanks, this worked! I’m curious though, what exactly is going on with this line:

```julia
Tree = @load RandomForestClassifier pkg=DecisionTree
tree = Tree()

```

The code works when I load RandomForestClassifier in that way, but doesn’t work when I do

```julia
using DecisionTree
tree = RandomForestClassifier()

```

EDIT: Here’s the error message I get:

```julia
MethodError: no method matching machine(::RandomForestClassifier, ::SubDataFrame{DataFrame, DataFrames.Index, Vector{Int64}}, ::CategoricalArrays.CategoricalVector{Bool, UInt32, Bool, CategoricalArrays.CategoricalValue{Bool, UInt32}, Union{}})
Closest candidates are:
  machine(!Matched::Type{<:Model}, ::Any...; kwargs...) at ~/.julia/packages/MLJBase/U4Dis/src/machines.jl:335
  machine(!Matched::Static, ::Any...; cache, kwargs...) at ~/.julia/packages/MLJBase/U4Dis/src/machines.jl:339
  machine(!Matched::Union{Symbol, Model}, ::Any, ::Any...; scitype_check_level, kwargs...) at ~/.julia/packages/MLJBase/U4Dis/src/machines.jl:363
  ...

Stacktrace:
 [1] top-level scope
   @ ~/code/julia/mltest/test_mlj.ipynb:6

```

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [November 22, 2022, 6:26pm UTC](https://discourse.julialang.org/t/how-to-properly-use-mljbase-roc-curve/90648/4 "2022-11-22T18:26:54Z")

</div>

I’m not an MLJ expert, but I think this is what the interface packages like `MLJDecisionTreeInterface` are for - they expose the models in a way that makes them usable with all the MLJ `machine`, `TunedModel`, `fit` etc machinery.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [December 12, 2022, 8:35pm UTC](https://discourse.julialang.org/t/how-to-properly-use-mljbase-roc-curve/90648/5 "2022-12-12T20:35:02Z")

</div>

For the record, there is this [manual entry](https://alan-turing-institute.github.io/MLJ.jl/dev/loading_model_code/) dealing with mode code loading.