# Setup topology with ClusterManagers

**URL:** <https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [April 19, 2021, 1:23pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591 "2021-04-19T13:23:40Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gregstrq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gregstrq/32/20620_2.png) [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Post date:** [April 19, 2021, 1:23pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/1 "2021-04-19T13:23:40Z")

</div>

How does one pass topology to a cluster manager? Julia docs mention the setup of topology in the case when `addprocs` dispatches on `machines`, but not on a cluster manager.

I am trying to do the following thing.

- I would like to create a job (with multiple processes) for each of the computing nodes I require via `addprocs(manager)`.
- For each of the jobs, I would like to have a master with a bunch of workers
- The parent process that ran `addprocs` should connect only to the master process in each of the nodes.

In essence, I want to create a “chain of command”: parent process would give instructions to the master processes on each of the nodes; these masters would give instructions to their respective workers.

The reason why I am trying to do that is the following: I need to share the structures between the workers from the same computing node.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [April 19, 2021, 2:29pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/2 "2021-04-19T14:29:53Z")

</div>

With `addprocs(manager)` you will create one master and add `n` workers. For a master-worker topology use the `topology` kwarg. For example, if using Slurm Manager then something like

```julia
addprocs(SlurmManager(n_workers), N=m_nodes, topology=:master_worker, exeflags="--project=.")

```

I don’t know if it’s currently possible to do a “chain of command” where you have the parent process spawn other “parent processes” that then spawn workers.

---

<div class="post-metadata">

**Author:** ![Gregstrq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gregstrq/32/20620_2.png) [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Post date:** [April 19, 2021, 2:43pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/3 "2021-04-19T14:43:33Z")

</div>

@affans, it seems that currently it is impossible to create workers from other workers.

On the other hand, I would simply like to change the topology. Master-worker topology implies that all the workers connect only to the master process which ran `addprocs`. I would like to have only one worker to be connected to the master process while the other workers connect to this worker.

I have looked at the docs, and it seems I don’t understand at all how does one changes topology.  
Can anyone point out some guide where it is discussed in detail how to setup custom topology? Some good examples?

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [April 19, 2021, 2:52pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/4 "2021-04-19T14:52:54Z")

</div>

> [@Gregstrq](#):
>
> I would like to have only one worker to be connected to the master process while the other workers connect to this worker.

I don’t think that’s currently possible with the available topologies. What you could do is set up a custom topology… from the documentation:

> The `launch` method of the cluster manager specifies the connection topology via fields `ident` and `connect_idents` in `WorkerConfig`. A worker with a cluster manager identity ident will connect to all workers specified in connect\_idents.

See [Distributed Computing · The Julia Language]ave([Distributed Computing · The Julia Language](https://docs.julialang.org/en/v1/stdlib/Distributed/)) You’ll probably have to read the source code for how the existing topologies are setup to properly understand how to implement a custom one.

---

<div class="post-metadata">

**Author:** ![Gregstrq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gregstrq/32/20620_2.png) [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Post date:** [April 19, 2021, 2:54pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/5 "2021-04-19T14:54:04Z")

</div>

Also, do I understand correctly that ClusterManagers imply that workers talk via ssh or TCP/IP connection?  
Is it possible to force the workers from the same computing node to talk directly to each other? (May be via some clever specification of topology?)

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [April 19, 2021, 2:56pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/6 "2021-04-19T14:56:03Z")

</div>

> [@Gregstrq](#):
>
> Also, do I understand correctly that ClusterManagers imply that workers talk via ssh or TCP/IP connection?

Yes.

> Is it possible to force the workers from the same computing node to talk directly to each other? (May be via some clever specification of topology?)

Again probably some custom topology but this is out of my expertise. I often just use the `master_worker` topology

---

<div class="post-metadata">

**Author:** ![Gregstrq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gregstrq/32/20620_2.png) [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Post date:** [April 19, 2021, 2:56pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/7 "2021-04-19T14:56:11Z")

</div>

> [@affans](#):
>
> You’ll probably have to read the source code for how the existing topologies are setup to properly understand how to implement a custom one.

That may be quite problematic. I hoped someone has bumped into this problem and has written a how-to guide…

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [April 19, 2021, 3:01pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/8 "2021-04-19T15:01:41Z")

</div>

`all_to_all` with `lazy=true` would work if the workers on each node only communicate on one another - see [here](https://docs.julialang.org/en/v1/manual/distributed-computing/#Specifying-Network-Topology-(Experimental)) and [here](https://docs.julialang.org/en/v1/stdlib/Distributed/#Distributed.addprocs).

---

<div class="post-metadata">

**Author:** ![Gregstrq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gregstrq/32/20620_2.png) [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Post date:** [April 19, 2021, 3:12pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591/9 "2021-04-19T15:12:48Z")

</div>

What type of connections does it use by default? Will it use different types of connection if two workers are on the same node or if they are on two different nodes?

Also, can someone clarify what are the available types of connections between workers?  
The docs mention ssh and TCP/IP. Do I understand correctly that these types are used when we have physically two different nodes connected either via InfiniBand or via internet?  
What kind of connection is used when the two workers are on the same node?