# Anyone has an implementation of generic boosting?

**URL:** <https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729>\
**Category:** Machine Learning\
**Created:** [February 6, 2021, 11:37am UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729 "2021-02-06T11:37:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 6, 2021, 11:37am UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729/1 "2021-02-06T11:37:06Z")

</div>

Dear All,

I would like to ask, as the title suggest, if anyone has a general implementation of boosting algorithm? I did some search and found that it is usually tightly coupled with a base learners being decision trees. But Boosting is a general meta-algorithm which assumes that the underlying base learner can fit possibly weighted samples and perform prediction on them.

I have recently started to wonder, why general people believes that NNs sucks on tabular datasets and Boosted decision trees shines. I came to conclusion that boosting might be the culprit, since single tree sucks as well. Since I would like to know, if I am right or wrong, I would like to test (and also would like to be right).

Due to my lack of time (you can also read it laziness), I would like to ideally hook already existing implementation (my six years old implementation is in matlab).

Well, thanks for answers and opinions on the matter of learning tabular data.

Tomas

---

<div class="post-metadata">

**Author:** ![tlienart](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlienart/32/7640_2.png) [@tlienart](https://discourse.julialang.org/u/tlienart)\
**Post date:** [February 6, 2021, 11:55am UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729/2 "2021-02-06T11:55:21Z")

</div>

There’s two unmaintained libs that might provide you with a decent starting point:

- [GitHub - svs14/GradientBoost.jl: Gradient boosting framework for Julia.](https://github.com/svs14/GradientBoost.jl)
- [GitHub - rakeshvar/AnyBoost.jl: A julia based machine learning package for boosting any loss, activation and constraint.](https://github.com/rakeshvar/AnyBoost.jl)

I don’t have experience with either so ymmv

PS: I think your assertion that “NNs suck on tabular datasets” is not up to date. Packages like e.g. AutoGluonTabular seem to suggest otherwise (though it blends NNs with other things).

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 6, 2021, 12:09pm UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729/3 "2021-02-06T12:09:36Z")

</div>

Thanks for links and correction of my knowlege. I was hoping that someone will point me to updated state of the art.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 6, 2021, 1:24pm UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729/4 "2021-02-06T13:24:24Z")

</div>

So I read the AutoGluonTabular, and it is not a model based purely on Neural Networks, but they use whatever model scikit learn offers, and an ensembling strategy seems to be a very important part of the solution.

Thanks a lot @tlienart for pointing me to this direction (more pointers are welcomed).

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 6, 2021, 6:20pm UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729/5 "2021-02-06T18:20:16Z")

</div>

I have fixed the GradientBoost, such that tests (almost) pass on 1.6. The only trouble is clashing of `fit!` and `predict` which I do not know, where they are defined.  
The fixed library is here  
[https://github.com/pevnak/GradientBoost.jl](https://github.com/pevnak/GradientBoost.jl)

I will try to contact the owner.

---

<div class="post-metadata">

**Author:** ![tlienart](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlienart/32/7640_2.png) [@tlienart](https://discourse.julialang.org/u/tlienart)\
**Post date:** [May 17, 2022, 12:48pm UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729/6 "2022-05-17T12:48:55Z")

</div>

I cloned your repo at d7fe4df to see if I could help with fit and predict but when trying to run the tests, most do not pass (with errors like `Util` or `GBBaseLearner` or `ML` not defined). Are you working on a separate branch?

> **Summary**
>
> fwiw I’m on 1.7 but I doubt that changes much here.
> 
> ```julia
> (GradientBoost) pkg> status
> Project GradientBoost v0.1.0
> Status `~/Desktop/tjd/GradientBoost.jl/Project.toml`
> [864edb3b] DataStructures v0.18.12
> [38e38edf] GLM v1.7.0
> [7f8f8fb0] LearnBase v0.4.1
> [30fc2ffe] LossFunctions v0.7.2
> [9920b226] MLDataPattern v0.5.5
> [872c559c] NNlib v0.8.5
> [429524aa] Optim v1.7.0
> [f2b01f46] Roots v2.0.1
> [a759f4b9] TimerOutputs v0.5.19
> [9a3f8284] Random
> [10745b16] Statistics
> 
> ```

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [December 16, 2024, 11:32am UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729/7 "2024-12-16T11:32:35Z")

</div>

Hi guys, @tlienart, @Tomas_Pevny.  
How far did you go with resurrection of the GBDT?

The `DecisionTree.jl` is great, but I’m so much surprised no finding general implementation of the boosted trees in Julia

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [December 16, 2024, 1:34pm UTC](https://discourse.julialang.org/t/anyone-has-an-implementation-of-generic-boosting/54729/8 "2024-12-16T13:34:30Z")

</div>

I sort of hacked the original implementation for my purposes, but always got tired. But to be honest, I gut it out, because I do not care about using GBDT, I wanted to Boost neural networks.