# New machine learning package, Julia implementation of XGBoost 

**URL:** <https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912>\
**Category:** Machine Learning\
**Created:** [February 7, 2018, 10:38pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912 "2018-02-07T22:38:49Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![statfactory](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statfactory/32/3510_2.png) [@statfactory](https://discourse.julialang.org/u/statfactory)\
**Post date:** [February 7, 2018, 10:38pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/1 "2018-02-07T22:38:49Z")

</div>

Hi,  
I am working on a new Julia machine learning package. What I have now is a pure Julia implementation of XGBoost in \< 500 lines of code:  
[https://github.com/Statfactory/JuML.jl](https://github.com/Statfactory/JuML.jl)

It seems to be as fast as original XGBoost but easier to use and with a smaller memory footprint. But don’t take my word for it:) See Readme for example usage with 10M obs airline dataset.

I am looking for feedback, testers, contributors etc

Cheers  
Adam

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 7, 2018, 11:04pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/2 "2018-02-07T23:04:30Z")

</div>

@oxinabox you should give it a spin in that blog post

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 7, 2018, 11:06pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/3 "2018-02-07T23:06:41Z")

</div>

Looking at the repo, you should add .jl to the name, add unit tests, setup CI, and get it registered.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [February 8, 2018, 12:53am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/4 "2018-02-08T00:53:19Z")

</div>

yeah, I may do a sequel, I also want to add GLMnet and Lasso.

I like XGboost, it is crazy effective.  
A pure julia implementations can only be a good thing.  
I was messing around the other day with giving it a custom loss function in python, and I was like "Now I have made it slow.', (further in python it actually changes the results even for the same loss function, presumably because rounding differences)

---

<div class="post-metadata">

**Author:** ![pasha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pasha/32/3319_2.png) [@pasha](https://discourse.julialang.org/u/pasha)\
**Post date:** [February 8, 2018, 1:22am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/5 "2018-02-08T01:22:25Z")

</div>

Nice. I like GLMnet with LASSO or Elasticnet too. Three suggestions beyond @ChrisRackauckas 's directions for how to make this a registered Julia package,

1. One of the things I liked about the GLMnet in R was `cv.glmnet`, which did a little cross validation to find a lambda. For XGBoost and all machine learning packages I feel that hyperparameter CV is useful.

2. I’m interested in individual models like this fit into machine learning frameworks like [MLBase.jl](https://github.com/JuliaStats/MLBase.jl) or systems like Caret on R. Because any model is only useful after multiple cross-validation.

3. In my prior job we used cox proportional hazards (survival) models a lot, and they didn’t always get included in an algorithm. GLMnet did (thank you Hastie) and I encourage you to do the same! It looks like this is [under consideration now for the xgboost C package](https://github.com/dmlc/xgboost/issues/386).

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [February 23, 2018, 4:13am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/6 "2018-02-23T04:13:11Z")

</div>

I was lucky to get a chance to chat to @statfactory as he was in town in Sydney. I tried the library and it worked flawlessly.

> [@oxinabox](#):
>
> I was messing around the other day with giving it a custom loss function in python, and I was like "Now I have made it slow.

This calls for a native Julia implementation of ML algorithms instead of relying on C++. I had always thought that the implementation is so hard that we need to just use the C++ but I can see now first hand what a skilled person can make of it in a short period of time! It’s really impressive!

There is definitely a commercial company in there. Incorporation into JuliaDB would be awesome, but an independent H2O but in pure Julia is a possibility!

---

<div class="post-metadata">

**Author:** ![statfactory](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statfactory/32/3510_2.png) [@statfactory](https://discourse.julialang.org/u/statfactory)\
**Post date:** [February 23, 2018, 8:39am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/7 "2018-02-23T08:39:11Z")

</div>

I have managed to improve performance and now it is 2x faster than C++ XGBoost with much smaller memory footprint. I am hoping to release it soon so stay tuned.

Adam

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [February 23, 2018, 11:52am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/8 "2018-02-23T11:52:26Z")

</div>

Just writing this down in case I forget

1. Julia allows customized functions and easy to modify codebase. For researchers, improvement or alternative implementation can be quickly trialled
2. Julia’s lazy programming and functional programming facilities allow low-cost creation of new columns
3. It is faster than the C++ implementation not because C++ is slow but because the functional style of programming which surfaces many optimizations that are otherwise hard to spot in large code-bases worked on by many people

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [February 23, 2018, 8:27pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/9 "2018-02-23T20:27:30Z")

</div>

> [@pasha](#):
>
> In my prior job we used cox proportional hazards (survival) models a lot, and they didn’t always get included in an algorithm. GLMnet did (thank you Hastie) and I encourage you to do the same!

I’m not sure if it helps but a pure julia implementation of Cox regression is available [here](https://github.com/ararslan/Survival.jl/blob/master/src/cox.jl#L134). It’s reasonably optimized (at least compared to matlab’s) but it’s single-threaded . I’m not sure what it would take to transform it into a XGBoost style implementation, but I figured having it as a reference could be better than starting from scratch.

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [September 18, 2018, 12:52am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/10 "2018-09-18T00:52:42Z")

</div>

Current xgboost doesn’t seem to work with either Julia 0.7 or 1.0. I am not able to compile it

---

<div class="post-metadata">

**Author:** ![Olof\_Salberger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/olof_salberger/32/4850_2.png) [@Olof\_Salberger](https://discourse.julialang.org/u/Olof_Salberger)\
**Post date:** [September 20, 2018, 9:20pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/11 "2018-09-20T21:20:15Z")

</div>

Gradient boosting is a really nice and hassle-free ML algorithm and beating XGboost in performance is really impressive. Something like this could be a killer app for Julia machine learning. It would be really cool to see every Kaggle contest being won by solutions that all either use or call Julia code.

---

<div class="post-metadata">

**Author:** ![rakeshvar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rakeshvar/32/3613_2.png) [@rakeshvar](https://discourse.julialang.org/u/rakeshvar)\
**Post date:** [September 20, 2018, 9:45pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/12 "2018-09-20T21:45:46Z")

</div>

Thanks. Just in time. I wanted to compare my algo to xgboost and having a pure julia implementation is great.

---

<div class="post-metadata">

**Author:** ![rakeshvar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rakeshvar/32/3613_2.png) [@rakeshvar](https://discourse.julialang.org/u/rakeshvar)\
**Post date:** [September 20, 2018, 11:21pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/13 "2018-09-20T23:21:15Z")

</div>

Here is some critical feedback:  
One of the main points of Gradient Boosting (and extensions like XGBoost) is its application/generalization to any loss (not just linear regression or logistic regression). This generalization is very easily achieved via Multiple Dispatch in `julia`. In that context, seeing that the package works only for logistic was surprising. Additionally, the code seems to be hard-coded around Logit loss.  
It would be great to abstract away the XGBoost algorithm from the direct dependence on loss, and instead, the algorithm can get the gradient (and hessian) from the loss Type (this is where multiple dispatch is so handy). This makes adding new losses very easy.  
Thanks.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [September 21, 2018, 12:31am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/14 "2018-09-21T00:31:51Z")

</div>

@rakeshvar Good points. I was wondering similar things. To help me better understand what goes on I am trying to implement a version that works.

I see a two pronged approach as appropriate. For qell known loss function (for which we know the derivatives), it better to have hard coded with convenient wrappers that pass in the loss and their loss. For novel loss functions, the code then passes in derivativea using auto differentaiton libraries.

E.g. two signatures `xgboost(....., loss = :logloss)` vs `xgboost(...., loss = some_novel_func)`

---

<div class="post-metadata">

**Author:** ![statfactory](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statfactory/32/3510_2.png) [@statfactory](https://discourse.julialang.org/u/statfactory)\
**Post date:** [September 21, 2018, 10:24am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/15 "2018-09-21T10:24:57Z")

</div>

I am the author of Julia XGBoost. Unfortunately my focus has now shifted to probabilistic machine learning (Bayesian networks) so I will not be able to develop this package any further.

Adam

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [September 21, 2018, 3:27pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/16 "2018-09-21T15:27:43Z")

</div>

Is this a request for contributors? 🙂

---

<div class="post-metadata">

**Author:** ![statfactory](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statfactory/32/3510_2.png) [@statfactory](https://discourse.julialang.org/u/statfactory)\
**Post date:** [September 21, 2018, 9:14pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/17 "2018-09-21T21:14:10Z")

</div>

You can develop it further on a fork if you want.

---

<div class="post-metadata">

**Author:** ![Joshua\_Bowles](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshua_bowles/32/4515_2.png) [@Joshua\_Bowles](https://discourse.julialang.org/u/Joshua_Bowles)\
**Post date:** [December 27, 2018, 1:51pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/18 "2018-12-27T13:51:57Z")

</div>

Curious what’s become of this project? I see there are a few forks but no work. Anyone know if this has been moved or adapted to a Juia working group or organization?

---

<div class="post-metadata">

**Author:** ![statfactory](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/statfactory/32/3510_2.png) [@statfactory](https://discourse.julialang.org/u/statfactory)\
**Post date:** [December 27, 2018, 2:52pm UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/19 "2018-12-27T14:52:08Z")

</div>

Anyone interested in contributing to the project? I can walk you through my implementation and give ideas for further development.

---

<div class="post-metadata">

**Author:** ![bpr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bpr/32/2710_2.png) [@bpr](https://discourse.julialang.org/u/bpr)\
**Post date:** [December 28, 2018, 7:32am UTC](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912/20 "2018-12-28T07:32:37Z")

</div>

I’m interested in contributing. I use XGBoost/Python at work, and I’d like to see a pure Julia version.

If you write down your ideas and a description on the github page, it will make it easier for contributors.

[Next page](https://discourse.julialang.org/t/new-machine-learning-package-julia-implementation-of-xgboost/8912.md?page=2)