# MLJ: selecting rows and columns for training in evaluate! for kernel regression

**URL:** <https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832>\
**Category:** General Usage\
**Tags:** mlj\
**Created:** [January 4, 2021, 4:51pm UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832 "2021-01-04T16:51:56Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pere](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pere/32/214250_2.png) [@Pere](https://discourse.julialang.org/u/Pere)\
**Post date:** [January 4, 2021, 4:51pm UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/1 "2021-01-04T16:51:57Z")

</div>

Hi everyone. This is my first post here, I’ve asked this on [stackoverflow](https://stackoverflow.com/questions/65346104/mlj-selecting-rows-and-columns-for-training-in-evaluate) but have had no luck. Perhaps someone can point me in the right direction.

I want to implement a kernel ridge regression that also works within MLJ. Moreover, I want to have the option to use either feature vectors or a predefined kernel matrix as in Python [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.kernel_ridge.KernelRidge.html).

When I run this code

```
const MMI = MLJModelInterface    
MMI.@mlj_model mutable struct KRRModel <: MLJModelInterface.Deterministic
        mu::Float64 = 1::(_ > 0)
        kernel::String = "linear"
end
function MMI.fit(m::KRRModel,verbosity::Int,K,y)
    K = MLJBase.matrix(K)
    fitresult = inv(K+m.mu*I)*y
    cache = nothing
    report = nothing
    return (fitresult,cache,report)
end

   
N = 10
K = randn(N,N)
K = K*K
a = randn(N)
y = K*a + 0.2*randn(N)
m = KRRModel()
kregressor = machine(m,K,y)
cv = CV(; nfolds=6, shuffle=nothing, rng=nothing)
evaluate!(kregressor, resampling=cv, measure=rms, verbosity=1)

```

the evaluate! function evaluates the machine on different subsets of rows of K. Due to the Representer Theorem, a kernel ridge regression has a number of nonzero coefficients equal to the number of samples. Hence, a reduced size matrix K[train\_rows,train\_rows] can be used instead of K[train\_rows,:].

To denote I’m using a kernel matrix I’d set `m.kernel = ""` . How do I make `evaluate!` select the columns as well as the rows to form a smaller matrix when `m.kernel = ""`?

This is my first time using MLJ and I’d like to make as few modifications as possible.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [January 4, 2021, 5:16pm UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/2 "2021-01-04T17:16:19Z")

</div>

I’m not too familiar with kernel ridge regression, but perhaps you can do something like this:

```julia
function MMI.fit(m::KRRModel, verbosity::Int, K, y)
    if m.kernel == ""
        K = # construct the smaller matrix
    else
        K = MLJBase.matrix(K)
    end
    fitresult = inv(K+m.mu*I)*y
    cache = nothing
    report = nothing
    return (fitresult, cache, report)
end

```

---

<div class="post-metadata">

**Author:** ![Pere](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pere/32/214250_2.png) [@Pere](https://discourse.julialang.org/u/Pere)\
**Post date:** [January 4, 2021, 11:17pm UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/3 "2021-01-04T23:17:53Z")

</div>

Thanks, this could work if I knew the indices of the chosen rows that are passed to fit!

The smaller matrix is built from choosing a subset of rows and columns and sampling their intersection. That is, if K is 10x10 and train\_idx=[1,4,6], then the smaller matrix is the 3x3 K[train\_idx,train\_idx].

Hence, since I need the indices I wonder whether I should define a new sampler, or whatever function passes the data to train!, to obtain the smaller square matrix to be used in training

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [January 4, 2021, 11:41pm UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/4 "2021-01-04T23:41:03Z")

</div>

Paging @ablaom.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [January 5, 2021, 11:31pm UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/5 "2021-01-05T23:31:20Z")

</div>

@Pere Thanks for your interest in MLJ.

The intended use of `evaluate!` is to estimate the generalisation error associated with some supervised learning model, by subsampling _observations_, as in cross-validation, a common use-case. I’m afraid there is no natural way for `evaluate!` do _feature_ subsampling.

[https://alan-turing-institute.github.io/MLJ.jl/dev/evaluating\_model\_performance/](https://alan-turing-institute.github.io/MLJ.jl/dev/evaluating_model_performance/)

FYI: There is a version of kernel regression implementing the MLJ model interface, namely kernel partial least squares regression from the package [https://github.com/lalvim/PartialLeastSquaresRegressor.jl](https://github.com/lalvim/PartialLeastSquaresRegressor.jl) .

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [January 6, 2021, 3:52am UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/6 "2021-01-06T03:52:56Z")

</div>

That said, if all you need is to generate `(train, test)` pairs of indices, according to some resampling strategy, such as `resampling=CV(nfolds=3)` you can call `MLJ.MLJBase.train_test_pairs`, as in

```julia
julia> resampling=CV(nfolds=3)
CV(
    nfolds = 3,
    shuffle = false,
    rng = Random._GLOBAL_RNG()) @377

julia> MLJBase.train_test_pairs(resampling, 1:10)
3-element Array{Tuple{Array{Int64,1},UnitRange{Int64}},1}:
 ([5, 6, 7, 8, 9, 10], 1:4)
 ([1, 2, 3, 4, 8, 9, 10], 5:7)
 ([1, 2, 3, 4, 5, 6, 7], 8:10)

```

---

<div class="post-metadata">

**Author:** ![Pere](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pere/32/214250_2.png) [@Pere](https://discourse.julialang.org/u/Pere)\
**Post date:** [January 7, 2021, 1:03pm UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/7 "2021-01-07T13:03:42Z")

</div>

I see, thanks for the help! I guess I’ll do the train/test loop as usual.

I’d actually already checked the partial LS package, but it calculates the kernel matrix each time the train/predict functions are called, something I want to avoid by using a precomputed matrix.

---

<div class="post-metadata">

**Author:** ![Pere](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pere/32/214250_2.png) [@Pere](https://discourse.julialang.org/u/Pere)\
**Post date:** [January 7, 2021, 3:52pm UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/8 "2021-01-07T15:52:21Z")

</div>

In case anyone comes across this, for now I’ve achieved what I wanted by setting the complete kernel matrix as a model parameter, which is ok for me since I’m using fairly small matrices. Then, the smaller kernel matrix is built inside fit/predict as @CameronBieganek suggested; the indices of the chosen column/rows are contained in what would be the matrix of features. Here’s the code for the kernel regressor:

```julia
using MLJModelInterface
using Random, LinearAlgebra
import MLJBase

const MMI = MLJModelInterface

MMI.@mlj_model mutable struct KRRModel <: MLJModelInterface.Deterministic
    mu::Float64 = 1::(_ > 0)
    kernel::String = ""
    kernel_matrix::Array{Float64,2} = zeros(1,1)
end

function MMI.fit(m::KRRModel,verbosity::Int,x,y)
    x = x[:]
    K = m.kernel_matrix[x,x]
    fitresult = (x,inv(K+m.mu*I)*y)
    cache = nothing
    report = nothing
    return (fitresult,cache,report)
end

function MMI.predict(::KRRModel, fitresult, xnew) 
    x = xnew[:]
    samples, coef = fitresult
    K = m.kernel_matrix[x,samples]
    return K*coef
end

function MMI.clean!(m::KRRModel)
    return 1
end

```

And the code to test it

```julia
N = 1000
svar = 0
K = randn(N,3)
K = K*K'
a = randn(N)
y = K*a + svar*randn(N)

cv = CV(; nfolds=6, shuffle=nothing, rng=nothing)

m = KRRModel(;mu=0.1e-8, kernel="", kernel_matrix = K)
x = MLJBase.matrix(collect(1:N)[:,:])
kregressor = machine(m,x,y)
evaluate!(kregressor, resampling=cv, measure=rms, verbosity=1)

```

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [January 22, 2021, 8:33am UTC](https://discourse.julialang.org/t/mlj-selecting-rows-and-columns-for-training-in-evaluate-for-kernel-regression/52832/9 "2021-01-22T08:33:19Z")

</div>

Nice. FYI `clean!` (if it is overloaded, which is optional) should return a string (a message explaining what has been mutated to enforce invariants for the struct). The fallback returns `""`.