# \[ANN\] LearnAPI.jl - Proposal for a basement-level machine learning API

**URL:** <https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048>\
**Category:** Package Announcements\
**Created:** [January 16, 2023, 8:42pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048 "2023-01-16T20:42:55Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![davidbp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidbp/32/463_2.png) [@davidbp](https://discourse.julialang.org/u/davidbp)\
**Post date:** [January 31, 2023, 4:26pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/22 "2023-01-31T16:26:13Z")

</div>

People might use the term `model` to refer to ‘abstract model’ and ‘trained model’. For example, according to [Microsoft documentation](https://learn.microsoft.com/en-us/windows/ai/windows-ml/what-is-a-machine-learning-model) model refers to learned model, but [Lux](http://lux.csail.mit.edu/stable/) uses `model` to refer to abstract model + Hyperparams.

I don’t mind Lux approach, using `Model` to refer to the abstract model (this includes hyperparameters) and `ModelArtifact` or `LearnedModel` to refer to the artifact needed to generate predictions (which contains learned parameters, weights, coefficients etc).

Usually `MLOps` people refer to `model` as the ‘abstract model that one can instanciate’ and [Model artifact](https://h2o.ai/wiki/artifacts/) to the file/files someone in production needs to productionize a model (which can be folder containing hyperparams, learned parameters and any metadata or information used to get predictions).

I have heard the use of `Model` to refer to `Abstract model` as well as `Learned model` but I have never seen the use of `model artifact` or `learned model` to refer to a model that has not been trained.

This seems to naturally yield to using concepts that define each other:

- `Model` vs `ModelArtifact`:

- `ModelHyperparams` vs `Model`:

Could the previous naming convention pairs (if used together in the same context) confuse anyone?

I like the design that @CameronBieganek proposed but I would not use `Options`, there is a word that is understood by everyone in ML which is `HyperParameters` or `HypeParams`.

Following this nomenclature I would have an implementation for ridge as follows:

```julia
using LearnAPI
using Tables

struct RidgeRegressorHyperparams
    lambda::Float64
end

RidgeRegressorHyperparams(; lambda=0.1) = RidgeRegressorHyperparams(lambda)

struct RidgeRegressor
    hyperparams::RidgeRegressorHyperparams
    coefs::Vector{Float64}  
    importances::Vector{Pair{Symbol, Float64}}
end

function LearnAPI.fit(Hyperparams::RidgeRegressorHyperparams, X, y; verbosity=0)
    x = Tables.matrix(X)
    s = Tables.schema(X)
    features = s.names

    coefs = (x'x + Hyperparams.lambda*I)\(x'y)

    importances = [features[j] => abs(coefs[j]) for j in eachindex(features)]
    reverse!(sort!(feature_importances, by=last))

    verbosity > 0 && @info "Features in order of importance: $(first.(feature_importances))"

    RidgeRegressor(Hyperparams, coefs, importances)
end

LearnAPI.predict(model::RidgeRegressor, Xnew) = Tables.matrix(Xnew) * model.coefs
LearnAPI.feature_importances(model::RidgeRegressor) = model.importances

```

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [February 1, 2023, 2:18pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/23 "2023-02-01T14:18:32Z")

</div>

> [@ExpandingMan](#):
>
> As a rather contrived example, suppose I have an input of NNN data points that consists of an m\times Nm×Nm\times N matrix AAA together with a set of n -mn−mn -m categorical features in a table and that the first step (perhaps a PCA or other dimensional reduction) is the multiplication MAMAMA. We really, really do not want to _break up_ AAA prior to the multiplication by, for example, allocating a new array for each of the NNN data points. Note that the most obvious implementation of `getobs` would return some kind of object where elements of the columns of AAA are mixed up with the other categorical features.

`StructArray(a=eachcol(A), b=[...], c=[...])` should represent this just fine. `View`s of such a structarray would preserve the structure.  
This could be a performance optimization: if a component is `Slices`, use optimized approach. At the same time, passing a StructArray or a regular array with one `Vector` for each element would work the same, just less efficiently. That’s convenient when you don’t want to assemble all `a`s into a single matrix manually.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [February 1, 2023, 7:18pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/24 "2023-02-01T19:18:45Z")

</div>

@aplavin Perhaps you meant to post this in another thread? I’m not understanding the relevance here.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [February 1, 2023, 8:42pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/25 "2023-02-01T20:42:24Z")

</div>

It’s a direct answer to the @ExpandingMan comment above, see the quotation.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [February 5, 2023, 3:03am UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/26 "2023-02-05T03:03:10Z")

</div>

Right, my mistake!

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [February 15, 2023, 6:46am UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/27 "2023-02-15T06:46:32Z")

</div>

@ExandingMan (and others). Just a quick acknowledgement of your LearnAPI.jl review which is most helpful and well-informed. I am busy with other things but rest assured this and other feedback are marinating at the back of my mind and I will return to this in due course.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [March 2, 2023, 10:35pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/28 "2023-03-02T22:35:03Z")

</div>

I’ve made a few changes to LearnAPI in response to some of the discussion:

- “model” structs are now called “algorithms”
- `predict` is now dispatched on the kind of target proxy (`LiteralTarget`, `Distribution`, `SurvivalFunction`, etc.)

Please see [this PR](https://github.com/JuliaAI/LearnAPI.jl/pull/17) for details.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [March 3, 2023, 3:58am UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/29 "2023-03-03T03:58:21Z")

</div>

Just 2c from me: I feel like “algorithms” isn’t a great fit, to me at least that’s more about the core mechanism at play than the configuration of it. Regarding the comment you made:

> After further thought (and I got as far as writing a new PR) I am less fond of “learner”. The problem is that plenty of machine learning algorithms don’t really learn (don’t generalize to new data).

I think there’s another aspect to this worth considering, the fact that the proposed package name is “LearnAPI”. In that respect “Learner” appears to be quite a good fit. I suppose you could switch to “AlgorithmsAPI” … but that seems unhelpfully broad.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [March 3, 2023, 7:37am UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/30 "2023-03-03T07:37:58Z")

</div>

I also dislike the term algorithm. I shared a comment in this thread explaining that a learning algorithm is a procedure like maximum likelihood estimation (MLE) that is used to learn the parameters of a learning model. These terms are well established in classical machine learning textbooks.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [March 5, 2023, 9:55pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/31 "2023-03-05T21:55:37Z")

</div>

> [@juliohm](#):
>
> I shared a comment in this thread explaining that a learning algorithm is a procedure like maximum likelihood estimation (MLE) that is used to learn the parameters of a learning model.

@juliohm I don’t understand how the use of “algorithm” in the updated docs is inconsistent with this view. It is, in fact, more inclusive, because there are algorithms in machine learning where there is no explicit statistical model (eg, DBSCAN clustering) but they still need a struct storing configuration parameters. As I see it, the change perfectly addresses the earlier objection to the use of “model” in the context of configuring an algorithm which is not itself a statistical model, only a means for learning one (or performing some other ML task).

@tecosaur Yes, I do agree: “Algorithm” sounds more about function than structure, and this is a compromise. Also, you don’t’ just _choose_ a configuration. You choose an algorithm _and_ a configuration of the algorithm. Ideally the word (or short composite word) should encapsulate both ideas. But it shouldn’t be a mouthful, or pose grammatical awkwardness in documentation. Nothing suggested so far meets all these criteria, as far as I can tell.

There have been opportunities for people to make alternative suggestions for the name of the configuration struct, here and in [this issue](https://github.com/JuliaAI/LearnAPI.jl/issues/12). With all due respect for the suggestions and engagement on this question, the only names I care to entertain now are “algorithm”, “strategy” or “learner”. (As noted in the issue, not all algorithms learn, ie generalize, so that is a minor drawback of the latter name).

**What do people prefer, “algorithm”, “strategy” or “learner”?** (edited)

If I can get 4 or more votes, I’m happy to go with the majority.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 5, 2023, 10:21pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/32 "2023-03-05T22:21:34Z")

</div>

I know this is not one of your preffered choices, but I still prefer `Options`. If I had to choose between `Algorithm` and `Learner`, I guess I would go with `Algorithm`.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [March 5, 2023, 10:27pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/33 "2023-03-05T22:27:09Z")

</div>

Of the two, I certainly prefer `Learner`. I must admit I do not relate to the concern that some instances will not be generalizing to new data—after all, I might “learn” a new vocab word every day, but that won’t necessarily help me generalize to knowing more words. I do relate to the concern that `Algorithm` is a pretty unintuitive term for a struct of configuration parameters. I usually think of an algorithm as having more to do with the actual implementation details of a particular learning process, and the hyperparameters define the search space for that learner.

That being said, I can _also_ relate to any concern that `Learner` might not be the most phonetically pleasing word.

> With all due respect for the suggestions and engagement on this question, the only names I care to entertain now

Well, ok 🙂 . If your mind ever changes on this, I suspect there is a word that can “encapsulate both algorithm and configuration” lurking behind a bit more brainstorm. Some unfiltered, possibly goofy, ideas:

- `Metamodel` or `Protomodel`
- `Schema` or `Modelschema`
- `Learnspace`
- `Blueprint`
- `Strategy` as suggested before
- There’s always `Confalgurithm` 😛

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 5, 2023, 11:05pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/34 "2023-03-05T23:05:54Z")

</div>

The word “configuration” has come up quite a few times in this thread. The abbrevation `config` is very commonly used in software development. So how about we just use the suffix `Config` for these structs? E.g. `RandomForestRegressorConfig`, `SupportVectorClassifierConfig`, etc.

Then, if desired, the documentation could refer to such an object as the “model configuration”.

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [March 5, 2023, 11:18pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/35 "2023-03-05T23:18:58Z")

</div>

@adienes I’d forgotten about my earlier proposal of “strategy” and have added that to the choices above.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [March 5, 2023, 11:25pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/36 "2023-03-05T23:25:42Z")

</div>

In that case, I think my ballot would read `Strategy ≻ Learner ≻ Algorithm` in that order

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 5, 2023, 11:36pm UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/37 "2023-03-05T23:36:05Z")

</div>

I guess I would go with `Strategy` if `Options` and `Config` are not on the table. 😛

---

<div class="post-metadata">

**Author:** ![wc4wc4wc4](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wc4wc4wc4/32/23038_2.png) [@wc4wc4wc4](https://discourse.julialang.org/u/wc4wc4wc4)\
**Post date:** [March 6, 2023, 7:51am UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/38 "2023-03-06T07:51:38Z")

</div>

`Config` or `Options` \> `Strategy` \> `Learner` \> `Algorithm`

---

<div class="post-metadata">

**Author:** ![DoktorMike](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/doktormike/32/2736_2.png) [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)\
**Post date:** [March 6, 2023, 8:23am UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/39 "2023-03-06T08:23:57Z")

</div>

Of the choices given: `Strategy`

---

<div class="post-metadata">

**Author:** ![davidbp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidbp/32/463_2.png) [@davidbp](https://discourse.julialang.org/u/davidbp)\
**Post date:** [March 6, 2023, 9:00am UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/40 "2023-03-06T09:00:47Z")

</div>

I still think adding and extraword (suffix) to `Model` might make reading documentation self evident about which one is just a struct and which one is “the whole object” from which one can make predictions. I haven’t heard anyone complaying that, if in the same Documentation finds any of the following pairs, would be confused by what they are:

- `Model` vs `ModelHyperParams`
- `Model` vs `ModelStrategy`
- `Model` vs `ModelStruct`
- `Model` vs `ModelConfig`
- `Model` vs `ModelBluePrint`
- `Model` vs `ModelSchema`

having two names that define each other (kind of `small` and `big`) tells you which one is a just a “schema/struct” vs object to use to generate predictions.

Of the proposed ones, for hyperparams struct I would use `Strategy`. But still, **`ModelStrategy`** for the struct and **`Model`** for the model instance would make it more evident which one is a model “ready to be used” and which one is a “configuration struct” for the model.

Arguing indefinitely about nomenclature is not super productive but I have to thank @ablaom for trying to get different points of view! At the end of the day is the smart way to build a community that will later use this API.

---

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [March 6, 2023, 9:05am UTC](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048/41 "2023-03-06T09:05:35Z")

</div>

After having read the current version of LearnAPI.jl I like the term “algorithm” ([“a precise step-by-step plan for a computational procedure”](https://en.wiktionary.org/wiki/algorithm)).

Some examples that I tried:

```julia
# Fitting a neural network classifier
alg = NeuralNetworkClassifier(builder = Short(n_hidden = 10), epochs = 3)
params, state, = fit(alg, 0, X, y)
y, = predict(alg, LiteralTarget, params, Xtest)
# continue training
alg.epochs = 10
params, state = update!(alg, 0, state, params, X, y)

# performing PCA
alg = PCA(variance_ratio = 1)
pcs, = fit(alg, 0, X)
scores = transform(alg, pcs, X2)

# hierarchical clustering
alg = HierarchicalClustering(k = 3, linkage = :complete)
_, _, report = fit(alg, 0, X)
plot(report.dendrogram)
report.cutter(h = 4) # returns cluster assignments

# Automatic Hyperparameter Tuning
alg = RidgeRegressor()
tuned_alg = TunedAlgorithm(algorithm = alg,
                           resampling = CV(nfolds = 10),
                           range = range(alg, :lambda,
                                         lower = 1e-9, upper = 1e-1))
params, _, report = fit(tuned_alg, 0, X, y)
y, = predict(tuned_alg, LiteralTarget, params, Xtest)

```

I would also be fine with `strategy` (I guess I would replace `alg` by `strat` in the examples above) or `learner`. Everything else, e.g. `config`, `options` or `hyper` would feel a bit awkward to me. Or would you be fine with `fit(options, 0, X, y)`?

[Previous page](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048.md?page=1)

[Next page](https://discourse.julialang.org/t/ann-learnapi-jl-proposal-for-a-basement-level-machine-learning-api/93048.md?page=3)