# MLJ and ClusterManagers

**URL:** <https://discourse.julialang.org/t/mlj-and-clustermanagers/51540>\
**Category:** Machine Learning\
**Tags:** parallel, mlj\
**Created:** [December 9, 2020, 5:51pm UTC](https://discourse.julialang.org/t/mlj-and-clustermanagers/51540 "2020-12-09T17:51:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![pfarndt](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@pfarndt](https://discourse.julialang.org/u/pfarndt)\
**Post date:** [December 9, 2020, 5:51pm UTC](https://discourse.julialang.org/t/mlj-and-clustermanagers/51540/1 "2020-12-09T17:51:06Z")

</div>

I want do some machine learning using `MLJ` on a cluster of 200+ nodes. To hook up my workers to the main process I want to use an `ElasticManager` from the `ClusterManagers` package (this way I can also utilize workers which come late because they had to queue in our queueing system).

I usually initiate my cluster workers and let the worker processes execute some code like

```julia
using ClusterManagers
using XYZ

elastic_worker(cookie, ip, port)

```

where `cookie`, `ip`, `port` have the correct values.

In general this scheme of “using-first-then-connect-worker” works and make the package XYZ available on the worker - this way I do not have to call an `@everywhere using XYZ` after my cluster is up and running - depending on `XYZ` and the number of nodes such an `@everywhere using XYZ` can actually take ages. If you dare try `@everywhere using DataFrames` on 200+ workers. Performance is miserable probably because the 200 workers want to load the very same files at the same time.

For a lot of packages this scheme works nicely. However when `XYZ` is `MLJ` (my preferred machine learning package):

```julia
using ClusterManagers
using MLJ
ClusterManagers.init_worker("oo")

```

I get an error:

```julia
**ERROR:** LoadError: AssertionError: isempty(PGRP.refs)

Stacktrace:

[1] **init_worker(** ::String, ::Distributed.DefaultClusterManager **)** at **/buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/cluster.jl:376**...

```

This is because also `MLJ` is trying to do something with workers and populates the `PGRP.refs`.

Is there a way to load `MLJ` but not initiate any parallel functionality?

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [December 13, 2020, 8:36pm UTC](https://discourse.julialang.org/t/mlj-and-clustermanagers/51540/2 "2020-12-13T20:36:10Z")

</div>

In [this slack thread](https://julialang.slack.com/archives/CC57ZE7EY/p1607670554313300?thread_ts=1607537684.312700&cid=CC57ZE7EY) @samuel_okon has suggested that the culprit is is a remote channel initialised in `MLJBase/src/ __init__.jl` . This channel is only used in testing - so there’s probably a work-around to get rid of it.

I’ve created a branch MLJBase#no-distributed with the channel initialisation removed. @pfarndt Can you see if this resolves your issue?

---

<div class="post-metadata">

**Author:** ![pfarndt](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@pfarndt](https://discourse.julialang.org/u/pfarndt)\
**Post date:** [December 14, 2020, 10:54am UTC](https://discourse.julialang.org/t/mlj-and-clustermanagers/51540/3 "2020-12-14T10:54:15Z")

</div>

yes - MLJBase#no-distributed resolves my issue - I hope you can get rid of this workaround. Thanks!

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [December 14, 2020, 10:33pm UTC](https://discourse.julialang.org/t/mlj-and-clustermanagers/51540/4 "2020-12-14T22:33:20Z")

</div>

Great, that’s good to hear

Opened an issue [https://github.com/alan-turing-institute/MLJBase.jl/issues/496](https://github.com/alan-turing-institute/MLJBase.jl/issues/496)

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [December 17, 2020, 8:16pm UTC](https://discourse.julialang.org/t/mlj-and-clustermanagers/51540/5 "2020-12-17T20:16:11Z")

</div>

Now sorted: [https://github.com/JuliaRegistries/General/pull/26575](https://github.com/JuliaRegistries/General/pull/26575)