# Changing the size of the \`target\` using MLJ's Learning Networks

**URL:** <https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002>\
**Category:** Machine Learning\
**Tags:** mlj\
**Created:** [March 1, 2024, 9:05am UTC](https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002 "2024-03-01T09:05:53Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![barakber](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barakber/32/207407_2.png) [@barakber](https://discourse.julialang.org/u/barakber)\
**Post date:** [March 1, 2024, 9:05am UTC](https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002/1 "2024-03-01T09:05:53Z")

</div>

Hi. I have a Static transformer `( <: Static )` that changes the size of the input (`X`). It’s using sliding windows, so that `size(X_tag)[1] = size(X)[1] - window_size`.

I’m trying to figure out what’s the right way to change the target variable’s size (`y`) using the MLJ’s Learning Network APIs.

I have a concrete problem with `evaluate!(...)`-ing the composed model, due to size mismatch of target variable

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [March 2, 2024, 1:47am UTC](https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002/2 "2024-03-02T01:47:41Z")

</div>

Thanks @barakber for giving MLJ’s learning networks a spin.

MLJBase/MLJ have a method, `selectrows`, which is overloaded for nodes, which you can therefore use immediately:

```julia
julia> y = source(rand(3))
Source @577 ⏎ `AbstractVector{Continuous}`

julia> ysmall = selectrows(y, 1:2)
Node @844
  args:
    1:	Source @577
  formula:
    #117(
      Source @577)

julia> ysmall()
2-element Vector{Float64}:
 0.4237311687983826
 0.6169364885950386

```

It works for any vector, matrix or table. Or, to do this “by hand”:

```julia
julia> ysmall = node(yy->yy[1:2], y)
Node @036
  args:
    1:	Source @577
  formula:
    #1(
      Source @577)

julia> ysmall()
2-element Vector{Float64}:
 0.4237311687983826
 0.6169364885950386

```

The `BalancedModel` wrapper from MLJBalancing reduces the length of input data, and is implemented using learning networks. That might be another example to look at, although here the oversamplers/undersamplers work on both `X` and `y` simultaneously (but are also `Static`).

This [Stacking tutorial](https://juliaai.github.io/DataScienceTutorials.jl/getting-started/stacking/) may provide further clues (although that tute contains some deprecated syntax).

If this is not sufficiently helpful, I may need a little more context.

---

<div class="post-metadata">

**Author:** ![barakber](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barakber/32/207407_2.png) [@barakber](https://discourse.julialang.org/u/barakber)\
**Post date:** [March 2, 2024, 1:47pm UTC](https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002/3 "2024-03-02T13:47:07Z")

</div>

thanks! @ablaom for the quick response.  
My problem is specifically with `evaluate!`-ing the composed model (for CV).

here is a simplified code example

```julia
module Example

using MLJBase
using RDatasets: dataset
using MLJXGBoostInterface

# ---
mutable struct ExampleTransformer <: Static
end

function MLJBase.transform(self::ExampleTransformer, _, X)
    selectrows(X, 1:size(X)[1] - 10)
end

# ---
mutable struct ExampleComposed <: ProbabilisticNetworkComposite
    transformer :: ExampleTransformer
    classifier :: XGBoostClassifier
end

function MLJBase.prefit(self::ExampleComposed, verbosity, X, y)
    Xs = source(X)
    ys = source(y)

    mach1 = machine(:transformer)
    X1 = transform(mach1, Xs)

    # --------
    # here I change the size of the `target`
    y1 = node((y -> y[1:end - 10]), ys) # <---
    # --------

    mach2 = machine(:classifier, X1, y1)
    yhat = predict(mach2, X1)
    return (;
        predict = yhat,
    )
end

function test()
    iris = dataset("datasets", "iris")
    y, X = unpack(iris, ==(:Species))
    y = categorical(map((x -> x == "setosa"), y))

    m = machine(
        ExampleComposed(
            ExampleTransformer(),
            XGBoostClassifier()
        ),
        X, y)

    m |> fit!

    # ==================================================
    # The error I get here -

    # ERROR: DimensionMismatch: Encountered two objects with sizes (15,) and (25,) which needed to match but don't.
    # ==================================================
    evaluate!(m, measure=auc)
end

end

```

the error that I get here is -

```julia
ERROR: DimensionMismatch: Encountered two objects with sizes (15,) and (25,) which needed to match but don't. 
Stacktrace:
  [1] check_dimensions
    @ ~/.julia/packages/MLJBase/ByFwA/src/utilities.jl:145 [inlined]
  [2] _check(measure::AreaUnderCurve, yhat::UnivariateFiniteVector{…}, y::CategoricalArrays.CategoricalVector{…})
    @ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/measures/measures.jl:60
  [3] Measure
    @ ~/.julia/packages/MLJBase/ByFwA/src/measures/measures.jl:132 [inlined]
  [4] value
    @ ~/.julia/packages/MLJBase/ByFwA/src/measures/measures.jl:202 [inlined]
  [5] value
    @ ~/.julia/packages/MLJBase/ByFwA/src/measures/measures.jl:196 [inlined]
  [6] (::MLJBase.var"#326#332"{…})(m::AreaUnderCurve, op::Function)
    @ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1237
  [7] #4
    @ ./generator.jl:36 [inlined]
  [8] iterate
    @ ./generator.jl:47 [inlined]
  [9] collect(itr::Base.Generator{Base.Iterators.Zip{Tuple{…}}, Base.var"#4#5"{MLJBase.var"#326#332"{…}}})
    @ Base ./array.jl:834
 [10] map(::Function, ::Vector{AreaUnderCurve}, ::Vector{typeof(predict)})
    @ Base ./abstractarray.jl:3409
 [11] fit_and_extract_on_fold
    @ ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1230 [inlined]
 [12] (::MLJBase.var"#307#308"{MLJBase.var"#fit_and_extract_on_fold#330"{…}, Machine{…}, Int64})(k::Int64)
    @ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1056
 [13] _mapreduce(f::MLJBase.var"#307#308"{…}, op::typeof(vcat), ::IndexLinear, A::UnitRange{…})
    @ Base ./reduce.jl:440
 [14] _mapreduce_dim
    @ ./reducedim.jl:365 [inlined]
 [15] mapreduce
    @ ./reducedim.jl:357 [inlined]
 [16] _evaluate!(func::MLJBase.var"#fit_and_extract_on_fold#330"{…}, mach::Machine{…}, ::CPU1{…}, nfolds::Int64, verbosity::Int64)
    @ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1055
 [17] evaluate!(mach::Machine{…}, resampling::Vector{…}, weights::Nothing, class_weights::Nothing, rows::Nothing, verbosity::Int64, repeats::Int64, measures::Vector{…}, operations::Vector{…}, acceleration::CPU1{…}, force::Bool, logger::Nothing, user_resampling::CV)
    @ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1259
 [18] evaluate!(::Machine{…}, ::CV, ::Nothing, ::Nothing, ::Nothing, ::Int64, ::Int64, ::Vector{…}, ::Vector{…}, ::CPU1{…}, ::Bool, ::Nothing, ::CV)
    @ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1335
 [19] evaluate!(mach::Machine{…}; resampling::CV, measures::Nothing, measure::AreaUnderCurve, weights::Nothing, class_weights::Nothing, operations::Nothing, operation::Nothing, acceleration::CPU1{…}, rows::Nothing, repeats::Int64, force::Bool, check_measure::Bool, verbosity::Int64, logger::Nothing)
    @ MLJBase ~/.julia/packages/MLJBase/ByFwA/src/resampling.jl:1015
 [20] test()
    @ Main.Example /workspaces/blab/src/Example.jl:59
 [21] top-level scope
    @ REPL[71]:1

```

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [March 4, 2024, 4:07am UTC](https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002/4 "2024-03-04T04:07:24Z")

</div>

That’s a nicely formulated example that quickly explains your issue very well, thank you.

The problem is that your new model `ExampleComposed` violates a basic assumption for use with `evaluate!`: if `yhat = predict(m, X)` then `yhat` must have the same number of observations as `X`. This is unavoidable, for how else is the routine supposed to know which ground truth observations are to be paired with which predictions?

You may want to rethink your approach, although one remedy comes to mind. You can alter your model so that the “missing” predictions are padded with `missing`. Then, provided you use a measure that supports `missing` predictions, you can use `evaluate!`. Unfortunately, `auc` does not support missing predictions (feel free to open an issue) so I’ve substituted `brier_score` in the following adaption of your example:

```julia
Pkg.activate(temp=true)
Pkg.add(["MLJBase", "RDatasets", "MLJModels", "StatisticalMeasures"])
using MLJBase, MLJModels
using RDatasets: dataset
using StatisticalMeasures

# ---
mutable struct ExampleTransformer <: Static
end

function MLJBase.transform(self::ExampleTransformer, _, X)
    selectrows(X, 1:size(X)[1] - 10)
end

model = ExampleTransformer()
mach = machine(model)

# ---
mutable struct ExampleComposed <: ProbabilisticNetworkComposite
    transformer
    classifier
end
composite = ExampleComposed(ExampleTransformer(), ConstantClassifier())

function MLJBase.prefit(self::ExampleComposed, verbosity, X, y)
    Xs = source(X)
    ys = source(y)

    mach1 = machine(:transformer)
    X1 = transform(mach1, Xs)

    # --------
    # here I change the size of the `target`
    y1 = node((y -> y[1:end - 10]), ys) # <---
    # --------

    mach2 = machine(:classifier, X1, y1)
    yshort = predict(mach2, X1)
    yhat = node(yshort) do y
        vcat(y, fill(missing, 10))
    end
    return (;
        predict = yhat,
    )
end

function test()
    iris = dataset("datasets", "iris");
    y, X = unpack(iris, ==(:Species));
    y = categorical(map((x -> x == "setosa"), y));

    m = machine(
        ExampleComposed(
            ExampleTransformer(),
            ConstantClassifier(),
        ),
        X, y)

    m |> fit!

    evaluate!(m, measure=brier_score)
end

```

**Caveat.** You should understand how StatisticalMeasures handles aggregation of missing values: the missing values are skipped, but they still contribute counts to normalisation. For example, the Brier scores above will be the sum of the non-missing observational scores, divided by 150, not 140.

---

<div class="post-metadata">

**Author:** ![barakber](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barakber/32/207407_2.png) [@barakber](https://discourse.julialang.org/u/barakber)\
**Post date:** [March 4, 2024, 7:31pm UTC](https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002/5 "2024-03-04T19:31:10Z")

</div>

thanks. so maybe I’m missing something about how `evaluate!` works in this situation.  
In my example `m |> fit!` works. It’s using `predict()`.

In my `prefit` I’m doing

```julia
    ...
    mach1 = machine(:transformer)
    X1 = transform(mach1, Xs)
    ...

```

which changes the number of observations (`size(X) - 10`), the same way I do for the target (`y`)

How is `evaluate!` different than `fit!` in this case?

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [March 7, 2024, 8:03pm UTC](https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002/6 "2024-03-07T20:03:37Z")

</div>

I’m simply saying that you cannot evaluate _any_ model (whether it comes from a learning network or whatever) if the assertion below can fail:

```julia
mach = machine(model, X, y) |> fit!
yhat = predict(mach, Xnew)
@assert nrows(yhat) == nrows(Xnew)

```

My prosal to pad output of `yhat ` with `missing`s, in the implementation of `model=ExampleComposedModel(...)`, resolves this particular problem. There may be another solution, but that’s one.

---

<div class="post-metadata">

**Author:** ![barakber](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barakber/32/207407_2.png) [@barakber](https://discourse.julialang.org/u/barakber)\
**Post date:** [March 7, 2024, 8:19pm UTC](https://discourse.julialang.org/t/changing-the-size-of-the-target-using-mljs-learning-networks/111002/7 "2024-03-07T20:19:51Z")

</div>

I see, thanks.  
So to me it sounds like these abstractions (`machine`, `fit!`, `predict`, `evaluate!` etc) are not well suited for changing the size (`nrows`) of `X` (and `y`).

My goal with Learning Networks, was to hide the transformer that changes `nrows` (it’s doing sliding windows), “under the hood”, and create a new model type that could be `fit!`ed, `predict`ed, `evaluate!`ed like any other model

In any case, thanks for your help!