# MLJ: accessing model fields from resampler

**URL:** <https://discourse.julialang.org/t/mlj-accessing-model-fields-from-resampler/53551>\
**Category:** Machine Learning\
**Created:** [January 18, 2021, 5:32pm UTC](https://discourse.julialang.org/t/mlj-accessing-model-fields-from-resampler/53551 "2021-01-18T17:32:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pere](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pere/32/214250_2.png) [@Pere](https://discourse.julialang.org/u/Pere)\
**Post date:** [January 18, 2021, 5:32pm UTC](https://discourse.julialang.org/t/mlj-accessing-model-fields-from-resampler/53551/1 "2021-01-18T17:32:02Z")

</div>

I have a composite model that has a parameter (alpha) that generates a probability distribution and then the features and samples are weighted according to this distribution. I also want the samples to be taken according to the parameter, and do all of this with a self-tuning model.

I’ve implemented this as a learning network as in this example in the [MLJ docs](https://alan-turing-institute.github.io/MLJ.jl/dev/composing_models/#Method-II:-Finer-control-(advanced)-1). The structure is:

> X,y → transformers → regressor → invert transform

The composite model has a parameter alpha to generate the weights for the transform. For the self-tuning, I also have a ResamplingStrategy depending on the same alpha. I’ve tried embedding the resampler as a field in the model and pass it to TunedModel as:

```julia
#LS <: ResamplingStrategy

r = range(lkrr_model, :(LS.alpha), lower=1, upper=10, scale=:log);
self_tuning_regressor = TunedModel(model=lkrr_model,
                              tuning=Grid(resolution=5),
                              resampling=lkrr_model.LS,
                              repeats=2,
                              range=r,
                              measure=rms);
tuned_kregressor = machine(self_tuning_regressor,K,y)
MLJ.fit!(tuned_kregressor)

```

However, every time alpha is changed, this is not reflected on the resampler. I guess this is because the model tuning function evaluates a clone of the model with the changed parameter. I can alter the resampler’s parameters at each call of `train_test_pairs` and the values are saved just fine, but I want it to be linked to the model so that I can autotune.

I have also tried setting the probabilities as weights and passing them around with a machine. But still, when calling `train_test_pairs(ls, rows, X, y, w`), the X, y and w are the ones associated with the self-tuning machine (`tuned_kregressor`), prior to any transformation. If I could access the data after the transformation then it would be ok.

How do I auto tune the resampler, or simply access the model’s fields from the resampler?

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [January 18, 2021, 8:30pm UTC](https://discourse.julialang.org/t/mlj-accessing-model-fields-from-resampler/53551/2 "2021-01-18T20:30:57Z")

</div>

I’m not sure I understand the motivation for having the resampling depend on model parameters. Normally resampling depends only on the data, not the particular model.

I don’t know of a way to achieve what you’re trying to do with the MLJ interface. It sounds like you need a custom tuning strategy, rather than a custom resampling stragegy. You could take a look at the interface provided in [MLJTuning.jl](https://github.com/alan-turing-institute/MLJTuning.jl). However, after briefly looking though their README, it doesn’t appear to me that MLJTuning supports what you’re trying to do, either.

Without knowing more about your model, it seems like the kind of sample weighting that you want should be occurring in the `fit` method rather than in `train_test_pairs`.

> [@Pere](#):
>
> I have also tried setting the probabilities as weights and passing them around with a machine. But still, when calling `train_test_pairs(ls, rows, X, y, w` ), the X, y and w are the ones associated with the self-tuning machine ( `tuned_kregressor` ), prior to any transformation. If I could access the data after the transformation then it would be ok.

Can you elaborate on why this approach does not work?

---

<div class="post-metadata">

**Author:** ![Pere](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pere/32/214250_2.png) [@Pere](https://discourse.julialang.org/u/Pere)\
**Post date:** [January 18, 2021, 9:38pm UTC](https://discourse.julialang.org/t/mlj-accessing-model-fields-from-resampler/53551/3 "2021-01-18T21:38:53Z")

</div>

> I’m not sure I understand the motivation for having the resampling depend on model parameters. Normally resampling depends only on the data, not the particular model.

Take for instance active sampling algorithms, where the model is first fitted for an initial set and then the inputs with the highest uncertainty are chosen to be labelled next. Here, the choice of next set of samples depends on the fit result. To implement this in a resampler you would have to either access the fit result or fit the model again (hence needing access to the hyperparameters).

> I don’t know of a way to achieve what you’re trying to do with the MLJ interface. It sounds like you need a custom tuning strategy, rather than a custom resampling stragegy. You could take a look at the interface provided in [MLJTuning.jl](https://github.com/alan-turing-institute/MLJTuning.jl). However, after briefly looking though their README, it doesn’t appear to me that MLJTuning supports what you’re trying to do, either.

I’d actually been looking at the MLJTuning code and I agree that I probably need a custom tuning strategy. Still, I’m not advanced enough yet for that so I was hoping there was an easier way. Alternatively, I won’t use MLJ’s autotuning and use something like [Hyperopt.jl](https://github.com/baggepinnen/Hyperopt.jl) instead since this would allow me to tune anything.

> Can you elaborate on why this approach does not work?

As in the network learner in the MLJ documentation, I have a transformation step and then a regressor which takes the transformed variables. Ideally, the transformer gives me the vector `w` that contains the sampling probabilities/weights. and I can do this

```julia
function MMI.fit(model::CompositeModel, verbosity, X, y,w=1)
    ys = source(y)
    Xs = source(X)
    yt = transform(model.lr,ys)
    Kt = transform(model.lr,Xs)
    ...
    yhat = inverse_transform(model.lr,zhat)
    ws = source(model.lr.weights)
    mach = machine(Deterministic(), Xs, ys, ws; predict=yhat)
    return!(mach, model, verbosity)
end

```

However, the autotuning function does not pass the transformed variables in `mach` to the resampler. It passes the X, y and w that are initially bound to the autotuning `machine(self_tuning_regressor,X,y,w)`. The transformed variables are only used in training the model.

Hope I made myself clear, thanks for the help!

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [January 19, 2021, 12:01am UTC](https://discourse.julialang.org/t/mlj-accessing-model-fields-from-resampler/53551/5 "2021-01-19T00:01:43Z")

</div>

It seems like you’re trying to fit a square peg into a round hole. 🙂

The resampling strategy interface in MLJ is designed for conducting model performance evaluation on fully trained models. (Although the resampling strategies sort of perform double duty, since you also have to pick one to use for `TunedModel`.) If you’re trying to do some sort of active sampling algorithm to learn your model, then I think it might make more sense to put all of that into the `fit` function. Even if training your model is an iterative process, the entire training process should normally be encapsulated in the `fit` function, so that the output of the `fit` function is a fully trained model.

Hope that helps!

---

<div class="post-metadata">

**Author:** ![Pere](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pere/32/214250_2.png) [@Pere](https://discourse.julialang.org/u/Pere)\
**Post date:** [January 19, 2021, 9:37pm UTC](https://discourse.julialang.org/t/mlj-accessing-model-fields-from-resampler/53551/6 "2021-01-19T21:37:40Z")

</div>

I ended up doing the sample selection all inside the `fit` function and not using a resampler (I simply pass all samples to the `fit` function). Now I can automatically tune the sampling distribution parameter as I wanted.

I’m actually porting some old code to the MLJ framework to more easily compare with other models and to learn, hence why I’m trying to do everything by the book. Thanks for the help! 😃