# \[ANN\] EvoLinear.jl for Linear Boosting

**URL:** <https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329>\
**Category:** Package Announcements\
**Tags:** machine-learning\
**Created:** [September 15, 2022, 6:48pm UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329 "2022-09-15T18:48:54Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [September 15, 2022, 6:48pm UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329/1 "2022-09-15T18:48:54Z")

</div>

A new package implementing linear boosting has just been launched: [EvoLinear.jl](https://github.com/jeremiedb/EvoLinear.jl).

It essentially covers the functionality provided by the `gblinear` learner found in [XGBoost](https://xgboost.readthedocs.io/en/stable/parameter.html#parameters-for-linear-booster-booster-gblinear).

Notably, it has L1 & L2 regularization, plus the following loss functions along their associated evaluation metrics:

- MSE (mean squared error)
- Logistic / Logloss
- Poisson
- Gamma
- Tweedie

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [September 15, 2022, 7:12pm UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329/2 "2022-09-15T19:12:24Z")

</div>

How is it different from [GitHub - svs14/GradientBoost.jl: Gradient boosting framework for Julia.](https://github.com/svs14/GradientBoost.jl) ?  
A general purpose boosting package, to which one can plug arbitrary classifier would be so nice.

---

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [September 15, 2022, 7:56pm UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329/3 "2022-09-15T19:56:15Z")

</div>

My perspective was to complement EvoTrees.jl with linear based learners, so most boosting needs are covered (trees + linear). I’m mainly concerned about implementation performance, along the flexibility for diverses loss functions.

I’m not sure if there are other important base learners that would be relevant in a boosting context?

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [September 16, 2022, 4:04am UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329/4 "2022-09-16T04:04:42Z")

</div>

What I meant was to just expose some api, such that one can add a different learner, for example neural networks instead of decision trees. But it is just a suggestion. My motivation was that having a go-to boosting package would be nice, especially if there are already few of those.

---

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [September 16, 2022, 3:23pm UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329/5 "2022-09-16T15:23:22Z")

</div>

Although I understand this was just used as an example, I’m not sure all kind of learners may be well adapted for boosting applications. I’d consider that a NN learns in a similar fashion to boosting, in the sense that weights are updated in sequence based on the current state of the underlying model. As such, my first impression would be that such boosting would turns out as a cumbersome and likely less efficient way to converge. Anecdotically, I was actually about to consider how effective deep learning optimizers can be so as train a linear booster using first order gradient rather than the second-order methods currently used by EvoLinear and XGBoost. Such approach would technically results in the NN model.  
Just some thoughts here, thanks for your input!

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [September 16, 2022, 3:40pm UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329/6 "2022-09-16T15:40:15Z")

</div>

Since EvoTrees already implements the MLJ model interface, wouldn’t making EvoLinear implement that as well solve this problem? Or do you mean substituting 1 tree → 1 NN in an existing boosting model?

---

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [September 18, 2022, 4:29am UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329/7 "2022-09-18T04:29:36Z")

</div>

I’m also planning to add MLJ integration to EvoLinear, likely within next week or so, once it gets added to the general registry.

---

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [September 20, 2022, 3:08am UTC](https://discourse.julialang.org/t/ann-evolinear-jl-for-linear-boosting/87329/8 "2022-09-20T03:08:31Z")

</div>

Starting with v0.3.0 [EvoLinear.jl](https://github.com/jeremiedb/EvoLinear.jl) now implements the MLJ interface.