# Addprocs() on remote machines failing

**URL:** <https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999>\
**Category:** Julia at Scale\
**Created:** [October 16, 2019, 8:03pm UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999 "2019-10-16T20:03:39Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![healyp](https://avatars.discourse-cdn.com/v4/letter/h/67e7ee/32.png) [@healyp](https://discourse.julialang.org/u/healyp)\
**Post date:** [October 16, 2019, 8:03pm UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999/1 "2019-10-16T20:03:39Z")

</div>

I would like to spawn some jobs on a cluster of linux machines. The machines are configured a little differently to my own so I need to specify where the julia executable is and the username when ssh’ing into them.

```julia
   p=map(x->(@sprintf("cs4115@cs305l-%02d.csis.ul.ie",x),:auto), [08,11])

```

When I make make the addprocs() call

```julia
    addprocs(p, topology=:master_worker, exename="/bin/julia", dir="/home/ug2018/cs4115")

```

I get the error

```julia
ERROR: IOError: connect: host is unreachable (EHOSTUNREACH)

```

I’m running v1.1.1 and the remote machines are running v1.0.4.

Thanks for any suggestions.

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [October 16, 2019, 8:36pm UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999/2 "2019-10-16T20:36:17Z")

</div>

> [@healyp](#):
>
> I’m running v1.1.1 and the remote machines are running v1.0.4.

I’m guessing this is the problem. I think you need workers to run the same Julia version.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [October 16, 2019, 8:47pm UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999/3 "2019-10-16T20:47:11Z")

</div>

Yes I’ve gotten this error before a few times. Indeed, you should run the same Julia version on all worker nodes/machines. Also, you should have passwordless ssh setup from the head node to the worker nodes.

---

<div class="post-metadata">

**Author:** ![healyp](https://avatars.discourse-cdn.com/v4/letter/h/67e7ee/32.png) [@healyp](https://discourse.julialang.org/u/healyp)\
**Post date:** [October 17, 2019, 6:03pm UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999/4 "2019-10-17T18:03:37Z")

</div>

OK, thanks a lot. That’s easy to fix.

But is there some deep reason why the versions must align? Just curious.

---

<div class="post-metadata">

**Author:** ![PatrickHealy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrickhealy/32/8887_2.png) [@PatrickHealy](https://discourse.julialang.org/u/PatrickHealy)\
**Post date:** [November 29, 2019, 12:42pm UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999/5 "2019-11-29T12:42:20Z")

</div>

> [@fredrikekre](#):
>
> > [@healyp](#):
> >
> > I’m running v1.1.1 and the remote machines are running v1.0.4.
> 
> I’m guessing this is the problem. I think you need workers to run the same Julia version.

Unfortunately, this doesn’t solve it for me:

```julia
julia> using Printf, Distributed

julia> versioninfo()
Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, sandybridge)
Environment:
  JULIAPATH = .:/home/healyp/src/julia-1.3.0/bin/
  JULIA_PKGDIR = /home/healyp/.julia/

julia> run(`ssh cs4115@cs305l-05.csis.ul.ie julia --version`)
julia version 1.3.0
Process(`ssh cs4115@cs305l-05.csis.ul.ie julia --version`, ProcessExited(0))

julia> params= (exename="/home/ug2018/cs4115/src/julia-1.3.0/bin/julia", dir="/home/ug2018/cs4115/pH")
(exename = "/home/ug2018/cs4115/src/julia-1.3.0/bin/julia", dir = "/home/ug2018/cs4115/pH")

julia> addprocs([("cs4115@cs305l-05.csis.ul.ie", 5)]; params...)
ERROR: TaskFailedException:
IOError: connect: host is unreachable (EHOSTUNREACH)
Stacktrace:
 [1] worker_from_id(::Distributed.ProcessGroup, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:1059
 [2] worker_from_id at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:1056 [inlined]
 [3] #remote_do#156 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:482 [inlined]
 [4] remote_do at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:482 [inlined]
 [5] kill at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/managers.jl:534 [inlined]
 [6] create_worker(::Distributed.SSHManager, ::WorkerConfig) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:581
 [7] setup_launched_worker(::Distributed.SSHManager, ::WorkerConfig, ::Array{Int64,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:523
 [8] (::Distributed.var"#43#46"{Distributed.SSHManager,Array{Int64,1},WorkerConfig})() at ./task.jl:333
Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] macro expansion at ./task.jl:319 [inlined]
 [3] #addprocs_locked#40(::Base.Iterators.Pairs{Symbol,Any,NTuple{5,Symbol},NamedTuple{(:tunnel, :sshflags, :max_parallel, :exename, :dir),Tuple{Bool,Cmd,Int64,String,String}}}, ::typeof(Distributed.addprocs_locked), ::Distributed.SSHManager) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:477
 [4] #addprocs_locked at ./none:0 [inlined]
 [5] #addprocs#39(::Base.Iterators.Pairs{Symbol,Any,NTuple{5,Symbol},NamedTuple{(:tunnel, :sshflags, :max_parallel, :exename, :dir),Tuple{Bool,Cmd,Int64,String,String}}}, ::typeof(addprocs), ::Distributed.SSHManager) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:441
 [6] #addprocs at ./none:0 [inlined]
 [7] #addprocs#243 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/managers.jl:118 [inlined]
 [8] (::Distributed.var"#kw##addprocs")(::NamedTuple{(:exename, :dir),Tuple{String,String}}, ::typeof(addprocs), ::Array{Tuple{String,Int64},1}) at ./none:0
 [9] top-level scope at REPL[9]:1

julia> 

```

Given that I can successfully ssh into the machine it seems like a strange error…

Thanks for any suggestions.

---

<div class="post-metadata">

**Author:** ![PatrickHealy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrickhealy/32/8887_2.png) [@PatrickHealy](https://discourse.julialang.org/u/PatrickHealy)\
**Post date:** [December 3, 2019, 11:11am UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999/6 "2019-12-03T11:11:20Z")

</div>

I posted previously about my difficulties with addprocs() on remote machines. I thought, perhaps, that the problem was with me ssh’ing in a) as a different user and, b) to a different directory structure. However, as the following demonstrates even attempting to add workers using ssh to login with defaults fails. Yet ssh appears to succeed. Does `Distributed.SSHManager` need some other information?

Thanks.

```julia
julia> using Distributed

julia> q=["cs144l-08", "cs144l-10"]
2-element Array{String,1}:
 "cs144l-08"
 "cs144l-10"

julia> map(p->run(`ssh $p julia --version`), q)
julia version 1.3.0
julia version 1.3.0
2-element Array{Base.Process,1}:
 Process(`ssh cs144l-08 julia --version`, ProcessExited(0))
 Process(`ssh cs144l-10 julia --version`, ProcessExited(0))

julia> addprocs(q) # fails
ERROR: TaskFailedException:
IOError: connect: host is unreachable (EHOSTUNREACH)
Stacktrace:
 [1] worker_from_id(::Distributed.ProcessGroup, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:1059
 [2] worker_from_id at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:1056 [inlined]
 [3] #remote_do#156 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:482 [inlined]
 [4] remote_do at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:482 [inlined]
 [5] kill at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/managers.jl:534 [inlined]
 [6] create_worker(::Distributed.SSHManager, ::WorkerConfig) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:581
 [7] setup_launched_worker(::Distributed.SSHManager, ::WorkerConfig, ::Array{Int64,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:523
 [8] (::Distributed.var"#43#46"{Distributed.SSHManager,Array{Int64,1},WorkerConfig})() at ./task.jl:333

...and 1 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] macro expansion at ./task.jl:319 [inlined]
 [3] #addprocs_locked#40(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:tunnel, :sshflags, :max_parallel),Tuple{Bool,Cmd,Int64}}}, ::typeof(Distributed.addprocs_locked), ::Distributed.SSHManager) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:477
 [4] #addprocs_locked at ./none:0 [inlined]
 [5] #addprocs#39(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:tunnel, :sshflags, :max_parallel),Tuple{Bool,Cmd,Int64}}}, ::typeof(addprocs), ::Distributed.SSHManager) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:441
 [6] #addprocs at ./none:0 [inlined]
 [7] #addprocs#243(::Bool, ::Cmd, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(addprocs), ::Array{String,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/managers.jl:118
 [8] addprocs(::Array{String,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/managers.jl:117
 [9] top-level scope at REPL[8]:1

julia> 

```

---

<div class="post-metadata">

**Author:** ![healyp](https://avatars.discourse-cdn.com/v4/letter/h/67e7ee/32.png) [@healyp](https://discourse.julialang.org/u/healyp)\
**Post date:** [December 9, 2019, 4:01pm UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999/7 "2019-12-09T16:01:39Z")

</div>

I believe that I have solved my particular issue.

Although I first didn’t think it applied, by calling`addprocs()` with `tunnel=true` I successfully set up a number of remote workers. It’s all there in the documentation 😉