# No variability in xgboost outputs? (XGBoost.jl)

**URL:** <https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211>\
**Category:** Statistics\
**Tags:** question\
**Created:** [August 11, 2021, 4:37pm UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211 "2021-08-11T16:37:06Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![matnbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matnbo/32/220166_2.png) [@matnbo](https://discourse.julialang.org/u/matnbo)\
**Post date:** [August 11, 2021, 4:37pm UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/1 "2021-08-11T16:37:06Z")

</div>

Dear all

I am using functions “xgboost” from the R packages “xgboost” and the  
Julia package “XGBoost.jl” that both interface the XGBoost library.  
May be I missed something  
but I observed surprising outputs from XGBoost.jl.  
When I fit one single tree with xgboost (Julia), there is  
no variability when I re-run the fitting on the same data,  
while I use only fractions of the data rows and columns for  
each run (subsample, colsample\_bytree), and even for each node of  
the tree (colsample\_bynode).

Since theses fraction selections are random,  
I expected to observe different results between the runs  
(note: I well observe this variability, as expected, with xgboost under R).

Below is a reproductible example:

```julia
n, p, m = 100, 100, 20
X = rand(n, p) ; y = rand(n) ;
Xnew = rand(m, p) ; ynew = rand(m) ; 

```

Fit of one single tree:

```julia
fm = xgboost(X, 1; label = y,
    booster = :gbtree,
    tree_method = :auto,
    num_parallel_tree = 1,
    subsample = .6,
    colsample_bytree = .8,
    colsample_bynode = 1/3,
    max_depth = 6,
    min_child_weight = 1,
    eta = 1,
    verbosity = 0)

```

Output:

```julia
pred = XGBoost.predict(fm, Xnew) ;
sum(ynew - pred).^2 / length(ynew) # MSEP

[1] train-rmse:0.237486
0.6847792450571403

```

When I re-run the fitting, I get the same result:

```julia
fm = xgboost(X, 1; label = y, ... Same as before)
pred = XGBoost.predict(fm, Xnew) ;
sum(ynew - pred).^2 / length(ynew) # MSEP

[1] train-rmse:0.237486
0.6847792450571403

```

I played with argument “seed\_per\_iteration” but this did not change:  
no variability between the runs. Did I miss something?

Actually, I have the same problem when I fit random forests or XGBoost models  
with Julia “xgboost” : I don’t observe variability between runs on the same  
data while I well observe variability with R xgboost (I used the  
same XGBoost parameterization under R and Julia)

```julia
fm = xgboost(X, 10; label = y,
    booster = :gbtree,
    tree_method = :auto,
    num_parallel_tree = 1,
    subsample = .8,
    colsample_bytree = .8,
    colsample_bynode = 1/3,
    max_depth = 6,
    min_child_weight = 1,
    eta = .3,
    verbosity = 0)

pred = XGBoost.predict(fm, Xnew) ;
sum(ynew - pred).^2 / length(ynew) # MSEP

[1] train-rmse:0.231707
[2] train-rmse:0.184608
[3] train-rmse:0.156864
[4] train-rmse:0.134460
[5] train-rmse:0.115025
[6] train-rmse:0.094434
[7] train-rmse:0.078340
[8] train-rmse:0.069557
[9] train-rmse:0.060034
[10] train-rmse:0.050421

0.425254884669798

```

When I re-run this code above, no variability in the results is observed.

[Another problem that I have is that “verbosity = 0” does not  
remove the printing information (from the doc, it should do,  
if I well understood). As shown in the example above,  
all round step information is printed anyway. (but this is also the case under R …).  
Does somebody know how to have a silent return?]

Thanks for any help

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [August 11, 2021, 9:38pm UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/2 "2021-08-11T21:38:01Z")

</div>

This looks like a genuine bug to me. I suggest you open an issue at XGBoost.jl.

BTW, if you’re interested in a pure julia gradient tree booster implementation, I suggest EvoTrees.j, which is actively maintained. Like XGBoost.jl and LighGBM.jl it has an MLJ interface.

---

<div class="post-metadata">

**Author:** ![matnbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matnbo/32/220166_2.png) [@matnbo](https://discourse.julialang.org/u/matnbo)\
**Post date:** [August 11, 2021, 9:49pm UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/3 "2021-08-11T21:49:16Z")

</div>

Thanks, I will open an issue at XGBoost.jl in few days.  
I know EvoTrees.jl )

---

<div class="post-metadata">

**Author:** ![matnbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matnbo/32/220166_2.png) [@matnbo](https://discourse.julialang.org/u/matnbo)\
**Post date:** [August 20, 2021, 10:22am UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/4 "2021-08-20T10:22:05Z")

</div>

Actually, I observe the same type of behaviour when I built a  
single tree with EvoTrees. It seems that the function returns  
always the same tree even if rowsample and colsample are set to \< 1.

```julia

n, p, m = 100, 100, 20

X = rand(n, p) ; y = rand(n) ;

Xnew = rand(m, p) ; ynew = rand(m) ; 

```

When I run several times:

```julia

pars = EvoTreeRegressor(loss = :linear,

           nrounds = 1, η = 1,

           rowsample = .5, colsample = .5,

           max_depth = 10,

           min_weight = 5.0) ;

fm = fit_evotree(pars, X, y) ;

pred = EvoTrees.predict(fm, Xnew)

sum(ynew - pred).^2 / length(ynew) # MSEP

```

I get always the same result. It looks like the same rows and columns are  
selected at each run of the code above.  
I observed the same behaviour when nrounds \> 1 (boosting): there is variability between  
the successive rounds, but not between the runs of the global code (which seems to indicate  
that the same rows and columns are selected between runs).

Is there a particular seed that is fixed inside the code of fit\_evotree?  
(I did not see specific arguments in the docs)

Or did I miss something?

---

<div class="post-metadata">

**Author:** ![matnbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matnbo/32/220166_2.png) [@matnbo](https://discourse.julialang.org/u/matnbo)\
**Post date:** [August 20, 2021, 11:14am UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/5 "2021-08-20T11:14:33Z")

</div>

By the way, I am also surprised by the fact that EvoTrees (which is faster  
than DecisionTree.jl) seems much slower to build trees than some R packages, such as  
“ranger” ([CRAN - Package ranger](https://cran.r-project.org/web/packages/ranger/index.html))

Below is an example with X(1000, 700) and y(1000)

Building a single regression tree with ranger in R:

```julia

library(ranger)

n <- 1000 ; p <- 700

X <- matrix(runif(n * p), ncol = p)

y <- runif(n)

```

```julia

dat <- data.frame(y, X)

system.time(

    fm <- ranger(y ~ ., data = dat, 

             num.trees = 1, 

             sample.fraction = 1, 

             mtry = 1, 

             max.depth = 10, 

             min.node.size = 5,

             replace = FALSE)

    )

```

```julia

utilisateur système écoulé 

       0.07 0.00 0.08 

```

(0.08 sec for one tree)

Same but with EvoTrees in Julia (I did run the code several times to remove  
the initial compilation time):

```julia

n, p = 1000, 700

X = rand(n, p) ; y = rand(n) ;

```

```julia

pars = EvoTreeRegressor(loss = :linear,

           nrounds = 1, η = 1,

           rowsample = 1, colsample = 1,

           max_depth = 10,

           min_weight = 5.0) ;

@time fm = fit_evotree(pars, X, y) ;

```

```julia

2.656099 seconds (2.28 M allocations: 3.712 GiB, 42.75% gc time)

```

Therefore almost 32 times slower.

When I did set max\_depth = 20, this became even worst: 0.12 sec for ranger, and  
my run with EvoTrees collapsed.

My surprise is that usually, Julia APIs are faster than R APIs. Again, may be  
I missed something.

(my config is: Intel(R) Core™ i9-10885H CPU @ 2.40GHz , RAM 32 Go)

---

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [August 24, 2021, 6:43am UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/6 "2021-08-24T06:43:25Z")

</div>

Regarding the variability, this can be tuned by using the `rng` parameter (which has been overlooked in the doc, thanks for pointing out):

```julia
pars = EvoTreeRegressor(loss = :linear,
           nrounds = 1, 
           η = 1,
           rowsample = 0.5, 
           colsample = 1,
           nbins = 16,
           max_depth = 10,
           min_weight = 5.0,
           rng=123);

```

Note that there’s also a `nbins` that has been specified here. EvoTrees always uses an histogram building approach, which is resulting in heavier preprocessing thank exact method, but with added speed benefit in the context of gradient boosting regression where typically dozens to several hundred trees will be built. It likely won’t be as optimal in a single regression tree context, nor where the number of observations is low.

This can be highlighted when splitting the preprocessing and the tree building part as follows:

```julia
julia> @time model, cache = EvoTrees.init_evotree(pars, X, y);
  1.078003 seconds (2.18 M allocations: 1.107 GiB, 34.28% gc time)

julia> @time EvoTrees.grow_evotree!(model, cache);
  0.073624 seconds (101.64 k allocations: 27.129 MiB)

```

It can be noticed that tree building per se is \< 0.1 sec, similar to what was observed with ranger.

Are those data dimensions (10K observations, 700 features) representative of your dataset size?

---

<div class="post-metadata">

**Author:** ![matnbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matnbo/32/220166_2.png) [@matnbo](https://discourse.julialang.org/u/matnbo)\
**Post date:** [August 24, 2021, 5:08pm UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/7 "2021-08-24T17:08:28Z")

</div>

Thanks for your answer Jeremie, and well noted the “rng” argument.  
BTW I have one question: argument “colsample” makes a sampling by tree or by node?

Yes I frequently work with such data size, and even more (e.g. 1,000 columns, or even \>10,000; n can also be \> or \>\> 1,000).

The origin of my post is that I was looking for a fast Random Forest (RF) tool under Julia, and I did not find any (DecisionTree.jl is for instance very slow for trees and RF, except for small data sizes). Therefore, I started to build my own bagging function and was looking for a fast tree builder.

There are now extremely fast RF tools under R, such as ranger or Rborist ([CRAN - Package Rborist](https://cran.r-project.org/web/packages/Rborist/index.html)). For instance, building a forest of 100 trees of depth 20 over X(1000, 700) takes ~0.3 sec with rangers:

```julia
n <- 1000 ; p <- 700
X <- matrix(runif(n * p), ncol = p)
y <- runif(n)

dat <- data.frame(y, X)
tic()
fm <- ranger(y ~ ., data = dat, 
             num.trees = 100, 
             sample.fraction = 1, # ==> bootstrap sampling
             mtry = 1/3, 
             max.depth = 20, 
             min.node.size = 5,
             replace = TRUE)
toc()

```

```julia
0.28 sec elapsed

```

I expected to find such tools in Julia. Sure it will come one day).

> It can be noticed that tree building per se is \< 0.1 sec, similar to what was observed with ranger

Yes, but if I well understood your code, the replication of the EvoTrees.init\_evotree step is necessary anyway when we want to build a forest.  
I also observed that when I increased max\_depth from 10 to 20 in EvoTreeRegressor, it collapsed Julia (even with nbins =16).

---

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [August 25, 2021, 3:43am UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/8 "2021-08-25T03:43:05Z")

</div>

> [@matnbo](#):
>
> argument “colsample” makes a sampling by tree or by node

Sampling if performed by tree.

I haven’t taken time yet to further benchmark cases where the number of features is quite large (like ~1000). Development of EvoTrees has originally been made around my kind of use cases with number of observations in the 100K to 10M range and number of features in the 50-250 range. As can be seen on the benchmark in the [README](https://github.com/Evovest/EvoTrees.jl#training), performance on such use cases is roughly competitive with XGBoost, which is a quite optimized implementation.

> [@matnbo](#):
>
> the replication of the EvoTrees.init\_evotree step is necessary anyway when we want to build a forest

Absolutely. My intention was to highlight that the reason for your EvoTrees experience being slow is more about data prep than tree building. That being said, in a RandomForest context, just like for boosting, if the number of trees is large, then this initialization step will become less relevant.  
Also, this initialization has not been as carefully optimized as the tree building step given its marginal impact in large boosted models. Therefore, there are some low hanging fruits on that side.

I’ll keep the option to add a RF option in EvoTrees has it should be fairly simple to integrate within current framework (likely not before a couple of weeks though!).

---

<div class="post-metadata">

**Author:** ![matnbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matnbo/32/220166_2.png) [@matnbo](https://discourse.julialang.org/u/matnbo)\
**Post date:** [August 25, 2021, 5:04pm UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/10 "2021-08-25T17:04:42Z")

</div>

**About Variability in XGBoost.jl:**

I finally found argument “seed”, to add in the body of the fonction xgboost and to vary (to get variability). It plays the same role as “rng” in EvoTrees.jl

---

<div class="post-metadata">

**Author:** ![matnbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matnbo/32/220166_2.png) [@matnbo](https://discourse.julialang.org/u/matnbo)\
**Post date:** [August 25, 2021, 5:10pm UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/11 "2021-08-25T17:10:59Z")

</div>

Thanks for the infos

> [@jeremiedb](#):
>
> with number of observations in the 100K to 10M range and number of features in the 50-250 range

I did look at the benchmark before using EvoTrees, but may be it would be useful to insist on this size specifity in the doc.

---

<div class="post-metadata">

**Author:** ![matnbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matnbo/32/220166_2.png) [@matnbo](https://discourse.julialang.org/u/matnbo)\
**Post date:** [August 25, 2021, 5:32pm UTC](https://discourse.julialang.org/t/no-variability-in-xgboost-outputs-xgboost-jl/66211/12 "2021-08-25T17:32:14Z")

</div>

**About Random Forests**

Actually I was wrong, XGboost already allows to fit RF models: [Random Forests(TM) in XGBoost — xgboost 2.0.0-dev documentation](https://xgboost.readthedocs.io/en/latest/tutorials/rf.html)  
for a very similar type of forest as, for instance, ranger.  
(a main difference is that ranger uses bootstrap to sample the observations, while XGBoost samples the observations without replacement, as usual in stochastic gradient boosting).

An example of RF syntax with XGBoost.jl

```julia
num_round = 1 ;
fm = xgboost(Xtrain, num_round; label = Float64.(vec(ytrain)),
           seed = Int64(round(rand(1)[1] * 10000)),
           booster = :gbtree,
           tree_method = :exact,
           num_parallel_tree = 50,
           subsample = .7,
           colsample_bytree = 1,
           colsample_bylevel = 1,
           colsample_bynode = 1/3,
           max_depth = 20,
           min_child_weight = 5,
           lambda = 0,
           eta = 1,
           verbosity = 0)

pred = XGBoost.predict(fm, Xtest)

```

On my data, I observed very similar prediction error rates between ranger and the RF models built from the R and Julia XGBoost packages, as well as for the computation times, e.g.: the three cited APIs used **~0.50 sec to build a forest of 50 trees of max\_depth = 20 over X(1100, 700)**, which is great and very encouraging ).