# Function to Calculate Kappa

**URL:** <https://discourse.julialang.org/t/function-to-calculate-kappa/72852>\
**Category:** Machine Learning\
**Tags:** machine-learning, mlj\
**Created:** [December 9, 2021, 7:06pm UTC](https://discourse.julialang.org/t/function-to-calculate-kappa/72852 "2021-12-09T19:06:58Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rahul](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rahul/32/30545_2.png) [@Rahul](https://discourse.julialang.org/u/Rahul)\
**Post date:** [December 9, 2021, 7:06pm UTC](https://discourse.julialang.org/t/function-to-calculate-kappa/72852/1 "2021-12-09T19:06:58Z")

</div>

Is there a function to calculate Kappa in MLJ? If no, I wrote a function (shown below) and I am looking for a more efficient implementation.

```julia
function kappa(predictions, original)
    # Confusion matrix 
    confmat = MLJ.confusion_matrix(mode.(predictions), original)
    confmat = confmat.mat
    z = (TP = confmat[1,1], FP = confmat[1,2], FN = confmat[2,1], TN = confmat[2,2])
    
    # Calculate kappa
    length_y = length(original)
    
    p_0 = (z.TN + z.TP)/length_y
    p_e_1 = (z.TP + z.FP) * (z.TP + z.FN)/(length_y^2)
    p_e_2 = (z.TN + z.FP) * (z.TN + z.FN)/(length_y^2)
    p_e = p_e_1 + p_e_2
    κ = (p_0 - p_e)/(1 - p_e)
    
    return κ
end

```

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [December 9, 2021, 7:12pm UTC](https://discourse.julialang.org/t/function-to-calculate-kappa/72852/2 "2021-12-09T19:12:17Z")

</div>

> [@Rahul](#):
>
> ```julia
> confmat = MLJ.confusion_matrix(mode.(predictions), original)
> 
> ```

From a quick look at their documentation it seems they don’t have a `MLJ.confusion_matrix!` method, so you are probably out of luck (or you would copy that method to do an in place update, how ugly;).

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [December 12, 2021, 9:30pm UTC](https://discourse.julialang.org/t/function-to-calculate-kappa/72852/3 "2021-12-12T21:30:52Z")

</div>

Thanks for the suggestion. Noted here: [https://github.com/JuliaAI/MLJBase.jl/issues/689](https://github.com/JuliaAI/MLJBase.jl/issues/689)

---

<div class="post-metadata">

**Author:** ![Rahul](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rahul/32/30545_2.png) [@Rahul](https://discourse.julialang.org/u/Rahul)\
**Post date:** [January 5, 2022, 5:09pm UTC](https://discourse.julialang.org/t/function-to-calculate-kappa/72852/4 "2022-01-05T17:09:37Z")

</div>

@ablaom Just an FYI, I wrote a function for multi-class.

```julia
function kappa(yhat, y)
    # Get confusion matrix
    try
        confmat = MLJ.confusion_matrix(mode.(yhat), y) #probabilistic
    catch
        confmat = MLJ.confusion_matrix(yhat, y) #deteministic
    end
    confmat = confmat.mat

    # sizes
    c = size(confmat)[1] # number of classes
    m = sum(confmat) # number of instances

    # relative observed agreement
    diags = [confmat[i, i] for i in 1:c]
    p_0 = sum(diags)/m

    # probability of agreement due to chance
    # for each class, this would be: (# positive predictions)/(# instances) * (# positive observed)/(# instances)
    p_e = 0
    for i in 1:c
        p_e_i = sum(confmat[i, j] for j in 1:c) * sum(confmat[j, i] for j in 1:c)/m^2
        p_e += p_e_i
    end

    # Kappa calculation
    κ = (p_0 - p_e)/(1 - p_e)

    return κ
end

```

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [January 5, 2022, 10:57pm UTC](https://discourse.julialang.org/t/function-to-calculate-kappa/72852/5 "2022-01-05T22:57:09Z")

</div>

Noted, thanks. The `try`/`catch` block is discouraged as slow. Basically `kappa` is a deterministic measure and would be implemented as such in MLJ. The user can compute `mode` if want to apply it to probabilistic predictions. In any case, MLJ’s `evaluate!` apparatus will allow you to specify deterministic measures where predictions are probabilistic, automatically calling the model’s `predict_mode` method instead of `predict` before passing on to the measure.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [January 5, 2022, 11:04pm UTC](https://discourse.julialang.org/t/function-to-calculate-kappa/72852/6 "2022-01-05T23:04:31Z")

</div>

BTW, a PR is welcome, if you are happy to include tests. Apart from the [user guidelines](https://alan-turing-institute.github.io/MLJ.jl/dev/performance_measures/) for measures, there are these [guidelines](https://github.com/JuliaAI/MLJBase.jl/blob/dev/src/measures/README.md) for adding new measures.

The new code would live [here](https://github.com/JuliaAI/MLJBase.jl/blob/dev/src/measures/finite.jl).

Further detail added at the issue: [https://github.com/JuliaAI/MLJBase.jl/issues/689#issuecomment-1006151903](https://github.com/JuliaAI/MLJBase.jl/issues/689#issuecomment-1006151903)