# Parallel Random Forest

**URL:** <https://discourse.julialang.org/t/parallel-random-forest/2401>\
**Category:** General Usage\
**Tags:** question\
**Created:** [March 2, 2017, 5:32am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401 "2017-03-02T05:32:44Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![rushirg](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@rushirg](https://discourse.julialang.org/u/rushirg)\
**Post date:** [March 2, 2017, 5:32am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/1 "2017-03-02T05:32:44Z")

</div>

I am using Random Forest algorithm for classification using `build_forest()` and then `apply_forest()`.  
As these operations are running on only one process, how could i parallelize these operations?  
And how can i generate graph for the same?

---

<div class="post-metadata">

**Author:** ![bicycle1885](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bicycle1885/32/107_2.png) [@bicycle1885](https://discourse.julialang.org/u/bicycle1885)\
**Post date:** [March 2, 2017, 6:50am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/2 "2017-03-02T06:50:28Z")

</div>

I’m not perfectly sure what you want to do, but I guess you want to parallelize training and prediction of random forest. The easiest way as far as I know is using `Threads.@threads`, which can run a loop body in parallel with multiple threads. An example code for training may looks like this:

```julia
function train_forest(X, Y, n_trees)
    trees = make_trees(n_trees)
    Threads.@threads for i in 1:n_trees
        train_tree!(trees[i], X, Y)
    end
    return RandomForest(trees)
end

```

---

<div class="post-metadata">

**Author:** ![rushirg](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@rushirg](https://discourse.julialang.org/u/rushirg)\
**Post date:** [March 2, 2017, 7:08am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/3 "2017-03-02T07:08:50Z")

</div>

Yes, what i am doing is building a forest mode like

```julia
model = build_forest(yTrain, xTrain, 20, 50, 1.0)

```

where yTrain is labels and xTrain is features and then applying the model

```julia
predTest = apply_forest(model, xTest)

```

xTest is test matrix

as all these operations are running on single process, what i want is to parallelize this task  
how could i do this?

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [March 2, 2017, 7:20am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/4 "2017-03-02T07:20:49Z")

</div>

I guess you are using the DecisionTree.jl package?

If so, it looks like the forest training is already parallelized through the `@parallel` macro, so you would only have to run `addprocs()` before training your model and then `build_forest` should use multiple workers (Check the link below for the package source code).

[https://github.com/bensadeghi/DecisionTree.jl/blob/master/src/regression.jl#L121](https://github.com/bensadeghi/DecisionTree.jl/blob/master/src/regression.jl#L121)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 2, 2017, 7:48am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/5 "2017-03-02T07:48:05Z")

</div>

Please mention that this is cross listed

[http://stackoverflow.com/questions/42548249/how-to-perform-parallel-execution-of-random-forest-in-julia](http://stackoverflow.com/questions/42548249/how-to-perform-parallel-execution-of-random-forest-in-julia)

---

<div class="post-metadata">

**Author:** ![rushirg](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@rushirg](https://discourse.julialang.org/u/rushirg)\
**Post date:** [March 5, 2017, 6:25pm UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/6 "2017-03-05T18:25:41Z")

</div>

Yes, I am using DecisionTree package, I tried `addprocs(4)` in my code. But, after reading the test data set, i got error like this:

 ![](https://global.discourse-cdn.com/julialang/original/3X/f/a/fa2c49a6a70e065bc3dd815789a2b7fc92ed5db3.png)

it is also saying error at `build_forest()` function call.  
also showing error like:

```julia
ERROR (unhandled task failure): On worker 4:

```

similarly for On worker 3

---

<div class="post-metadata">

**Author:** ![bjarthur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bjarthur/32/9638_2.png) [@bjarthur](https://discourse.julialang.org/u/bjarthur)\
**Post date:** [March 6, 2017, 1:41pm UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/7 "2017-03-06T13:41:24Z")

</div>

after addprocs you need to load your packages on each worker, like this:

import DecisionTree  
@everywhere using DecisionTree

---

<div class="post-metadata">

**Author:** ![rushirg](https://avatars.discourse-cdn.com/v4/letter/r/e8c25b/32.png) [@rushirg](https://discourse.julialang.org/u/rushirg)\
**Post date:** [March 6, 2017, 4:49pm UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/8 "2017-03-06T16:49:42Z")

</div>

Thank you @bjarthur 😄 it works !

But now i am getting less accuracy than previously on one process. is there any way to improve accuracy and efficiency of the algorithm?

Also, is there any way to store trained model so that i can load it and directly used it on test data set. As every time it is training the model.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [March 6, 2017, 5:24pm UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/9 "2017-03-06T17:24:42Z")

</div>

> [@rushirg](#):
>
> But now i am getting less accuracy than previously on one process. is there any way to improve accuracy and efficiency of the algorithm?

That’s probably just a random fluctuation. You can improve accuracy by tweaking the hyperparameters (depth, number of trees, pruning threshold), but you have to be careful about overfitting. You can either setup cross-validation yourself and do a loop over different combinations of hyperparameter values, or use the [ScikitLearn.jl interface](https://github.com/bensadeghi/DecisionTree.jl#scikitlearnjl), along with [GridSearchCV to do model selection](http://scikitlearnjl.readthedocs.io/en/latest/model_selection/).

> [@rushirg](#):
>
> Also, is there any way to store trained model so that i can load it and directly used it on test data set. As every time it is training the model.

[JLD.jl](https://github.com/JuliaIO/JLD.jl) should work for saving pure-Julia structures to disk.

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [September 20, 2018, 5:21am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/10 "2018-09-20T05:21:35Z")

</div>

How can I do parrallel computing with modules imported using PyCall and @pyimport. I am trying to do something like this but it does not work. If I make n\_jobs \>1, and remove 3rd line of code from top (@everywhere (@pyimport lightgbm as lgb) ) it still uses single processor. I am using julia 1.0 on win 10 64 bit.

```julia
using PyCall : @pyimport
@pyimport lightgbm as lgb
@everywhere (@pyimport lightgbm as lgb) 
        model = lgb.LGBMClassifier(colsample_bytree=1.0,
                    learning_rate=0.1, max_depth=-1, min_child_samples=20,
                    min_child_weight=0.001, min_split_gain=0.0, n_estimators=250,
                    n_jobs=1, num_leaves=31, objective="binary", random_state=123,
                    reg_alpha=0.0, reg_lambda=0.0, subsample=1.0)
        
        fit!(model, X, y)

```

The error displayed is

```julia
On worker 6:
LoadError: UndefVarError: @pyimport not defined
top-level scope
eval at .\boot.jl:319
#116 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:276
run_work_thunk at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:56
run_work_thunk at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
in expression starting at In[37]:13
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:407
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:398
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:419
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:419
(::getfield(Distributed, Symbol("##163#165")){Module,Expr})() at .\task.jl:259

...and 3 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:226
 [2] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\macros.jl:207
 [3] top-level scope at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\macros.jl:190
 [4] top-level scope at In[37]:5

```

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [September 20, 2018, 6:07am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/11 "2018-09-20T06:07:40Z")

</div>

maybe it helps if you do  
`@everywhere using PyCall`  
such that the package is loaded on all procs

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [September 20, 2018, 7:22am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/12 "2018-09-20T07:22:52Z")

</div>

@bernhard  
Thanks it does starts parrallel processing. However, I am not gaining any improvement in speed by doing this. I have data size of about ( 220K,28). Without parallel processing one single run takes about 10.6s. With parallel processing feature on, it takes about 16s.

I would like to know how can I gain improvement in speed?

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [September 20, 2018, 7:27am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/13 "2018-09-20T07:27:06Z")

</div>

Well, parallelization is not always trivial. Not all problems benefit from it (it depends on cache, data size, …)

However, I am not quite sure what exactly you are running in your code because you are using PyCall.  
Effectively the model is fitted in Python, right?  
If so, I am not sure if any ‘Distributed Code in Julia’ (or additional julia procs) will change anything at all, because python is doing the work here.

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [September 20, 2018, 7:42am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/14 "2018-09-20T07:42:01Z")

</div>

@bernhard  
I have recently moved to Julia and trying to move my Python models to Julia. Since there is no lightgbm or XGBoost model in julia, I am trying to use the same thorugh PyCall in Julia.

I agree that calling python models in Julia may not be as efficient as native Julia models but I have no choice and hence my current exploration. In Python with the help of Cython, I was able to run 10 iterations of 3-fold CV in approx 95 seconds that too using single core.

Just another question. If I were to warp the whole model fit in a function, what changes would I need to make in the code (given in my first post).

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [September 20, 2018, 7:58am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/15 "2018-09-20T07:58:11Z")

</div>

Hm, I do not know if there is a native xgboost or gbm implementation in Julia. It does not seem to be the case. Maybe you can find something on [https://pkg.julialang.org/](https://pkg.julialang.org/)  
But I guess you already searched and opted for PyCall (an alternative would of course by RCall).

Have you tried this: [https://github.com/bensadeghi/DecisionTree.jl](https://github.com/bensadeghi/DecisionTree.jl) ?  
If your problem is binary, classification might work right?

I have an **experimental** Julia package which includes a regression boosting approach, but it is not well documented and **may likely not** fit your purpose ([GitHub - kafisatz/DecisionTrees.jl: Julia Decision Tree Algorithms for Regression](https://github.com/kafisatz/DecisionTrees.jl) )

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [September 20, 2018, 8:00am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/16 "2018-09-20T08:00:04Z")

</div>

> [@Ajaychat3](#):
>
> Just another question. If I were to warp the whole model fit in a function, what changes would I need to make in the code (given in my first post).

Not sure what you mean by that  
how about this:

```julia
function myfit(X,y)
model = lgb.LGBMClassifier(colsample_bytree=1.0,
                    learning_rate=0.1, max_depth=-1, min_child_samples=20,
                    min_child_weight=0.001, min_split_gain=0.0, n_estimators=250,
                    n_jobs=1, num_leaves=31, objective="binary", random_state=123,
                    reg_alpha=0.0, reg_lambda=0.0, subsample=1.0)
        
        fit!(model, X, y)
return model
end

```

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [September 20, 2018, 8:58am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/17 "2018-09-20T08:58:03Z")

</div>

Thanks for your quick revert.

```julia
Not sure what you mean by that

```

What I meant was that what changes would be required for using parrallel computing inside a function call to the model being fit. You have already shared a code for wrapping the model fit inside the function call. Where and what code do I put inside or outside the code to use parrallel computing in this case.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [September 20, 2018, 9:06am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/18 "2018-09-20T09:06:08Z")

</div>

> **[GitHub - dmlc/XGBoost.jl: XGBoost Julia Package](https://github.com/dmlc/XGBoost.jl)**
>
> XGBoost Julia Package. Contribute to dmlc/XGBoost.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [September 20, 2018, 9:11am UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/19 "2018-09-20T09:11:07Z")

</div>

I have looked at it but not able to compile the model.

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [September 20, 2018, 3:13pm UTC](https://discourse.julialang.org/t/parallel-random-forest/2401/20 "2018-09-20T15:13:10Z")

</div>

Thanks I will look into the models suggested by you.

[Next page](https://discourse.julialang.org/t/parallel-random-forest/2401.md?page=2)