# Building cluster for Julia parallel computations

**URL:** <https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848>\
**Category:** New to Julia\
**Tags:** parallel\
**Created:** [May 22, 2017, 11:06am UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848 "2017-05-22T11:06:57Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivborissov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivborissov/32/2841_2.png) [@ivborissov](https://discourse.julialang.org/u/ivborissov)\
**Post date:** [May 22, 2017, 11:06am UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/1 "2017-05-22T11:06:57Z")

</div>

Hi ,  
What is the best way to set up cluster using local lab PCs for parallel Julia computations?  
In our lab we have hybrid environment (Windows-Linux) dominated by Windows servers.  
I have seen ClusterManagers and MPI libraries, but ClusterManagers jl seems to support Linux environment and MPImanager is said to support Julia 0.4 only.  
Could you please give me some advice for this use-case?

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [May 22, 2017, 5:18pm UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/2 "2017-05-22T17:18:09Z")

</div>

This may not be the most elegant solution, but it worked for me in the past. I set up the ssh capability between the computers and added the procs like so:

```julia
addprocs(["myComputer1@IP1","myComputer2@IP2", "myComputer2@IP3"],dir="/Applications/Julia-0.4.0.app/Contents/Resources/julia/bin/")

```

Each entry is the computer name followed by the IP address. dir points to the file that launches Julia. If its in a different location on each computer, it might be possible to pass a vector of locations.

---

<div class="post-metadata">

**Author:** ![ivborissov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivborissov/32/2841_2.png) [@ivborissov](https://discourse.julialang.org/u/ivborissov)\
**Post date:** [May 23, 2017, 9:37am UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/3 "2017-05-23T09:37:15Z")

</div>

Christopher, thanks!  
I am trying to use my Linux machine as master process and start workers on remote Windows using Bitvise ssh server. I can see that ssh connection is established but nothing is started. Trying different types of command syntax.

```julia
addprocs(["ivan@win-julia2"], dir="\Julia-0.5.2\bin",exename="julia.exe")

  <event seq="216" time="2017-05-23 00:58:15.282668 +0300" app="BvSshServer 7.31" name="I_EXECS_COMMAND_EXECUTED" desc="Command executed.">
    <session id="1024" service="SSH" remoteAddress="192.168.56.101:48922" windowsAccount="WIN-JULIA2\ivan"/>
    <channel type="session" id="1"/>
    <parameters command="cmd.exe /c sh -l -c &quot;cd 'Julia-0.5.2\bin' &amp;&amp; julia.exe --worker zT0oLTnPGTJNL85n&quot;" initDir="C:\Users\ivan" execRequest="sh -l -c &quot;cd 'Julia-0.5.2\bin' &amp;&amp; julia.exe --worker zT0oLTnPGTJNL85n&quot;"/>
  </event>

  <event seq="217" time="2017-05-23 00:58:15.321424 +0300" app="BvSshServer 7.31" name="I_CHANNEL_SESSION_CLOSED" desc="Session channel closed.">
    <session id="1024" service="SSH" remoteAddress="192.168.56.101:48922" windowsAccount="WIN-JULIA2\ivan"/>
    <channel type="session" id="1"/>
  </event>

  <event seq="218" time="2017-05-23 00:58:15.323793 +0300" app="BvSshServer 7.31" name="I_SESSION_DISCONNECTED_NORMALLY" desc="Session disconnected normally.">
    <session id="1024" service="SSH" remoteAddress="192.168.56.101:48922" windowsAccount="WIN-JULIA2\ivan"/>
    <parameters disconnectReason="SshError"/>
    <error type="Flow" component="SshManager/transport" class="RemoteSshDisconn" code="ByApplication" description="disconnected by user"/>
  </event>
</log>

```

---

<div class="post-metadata">

**Author:** ![ivborissov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivborissov/32/2841_2.png) [@ivborissov](https://discourse.julialang.org/u/ivborissov)\
**Post date:** [May 23, 2017, 10:13am UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/4 "2017-05-23T10:13:26Z")

</div>

```julia
  <event seq="216" time="2017-05-23 00:58:15.282668 +0300" app="BvSshServer 7.31" name="I_EXECS_COMMAND_EXECUTED" desc="Command executed.">
    <session id="1024" service="SSH" remoteAddress="192.168.56.101:48922" windowsAccount="WIN-JULIA2\ivan"/>
    <channel type="session" id="1"/>
    <parameters command="cmd.exe /c sh -l -c &quot;cd 'Julia-0.5.2\bin' &amp;&amp; julia.exe --worker zT0oLTnPGTJNL85n&quot;" initDir="C:\Users\ivan" execRequest="sh -l -c &quot;cd 'Julia-0.5.2\bin' &amp;&amp; julia.exe --worker zT0oLTnPGTJNL85n&quot;"/>
  </event>

  <event seq="217" time="2017-05-23 00:58:15.321424 +0300" app="BvSshServer 7.31" name="I_CHANNEL_SESSION_CLOSED" desc="Session channel closed.">
    <session id="1024" service="SSH" remoteAddress="192.168.56.101:48922" windowsAccount="WIN-JULIA2\ivan"/>
    <channel type="session" id="1"/>
  </event>

  <event seq="218" time="2017-05-23 00:58:15.323793 +0300" app="BvSshServer 7.31" name="I_SESSION_DISCONNECTED_NORMALLY" desc="Session disconnected normally.">
    <session id="1024" service="SSH" remoteAddress="192.168.56.101:48922" windowsAccount="WIN-JULIA2\ivan"/>
    <parameters disconnectReason="SshError"/>
    <error type="Flow" component="SshManager/transport" class="RemoteSshDisconn" code="ByApplication" description="disconnected by user"/>
  </event>
</log>

```

---

<div class="post-metadata">

**Author:** ![tkelman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkelman/32/692_2.png) [@tkelman](https://discourse.julialang.org/u/tkelman)\
**Post date:** [May 24, 2017, 5:34am UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/5 "2017-05-24T05:34:39Z")

</div>

The remote workers are trying to run `sh -l -c ...` which is unix-specific. Would need some patches to make it capable of working with Windows workers.

---

<div class="post-metadata">

**Author:** ![ivborissov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivborissov/32/2841_2.png) [@ivborissov](https://discourse.julialang.org/u/ivborissov)\
**Post date:** [May 26, 2017, 9:51pm UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/6 "2017-05-26T21:51:19Z")

</div>

Thanks, exactly. Now I am thinking of using clustermanagers with htcondor instead of ssh. However condor jl also requires customization to support windows nodes)

---

<div class="post-metadata">

**Author:** ![ivborissov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivborissov/32/2841_2.png) [@ivborissov](https://discourse.julialang.org/u/ivborissov)\
**Post date:** [June 8, 2017, 11:10am UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/7 "2017-06-08T11:10:38Z")

</div>

Finally I have installed htcondor and I am trying to start remote workers through condor\_submit. I have reconfigured condor.jl to run on windows but I have encountered an error with the way workers are registered.

Workers are started on remote computers with the following .cmd

```julia
cd C:\Julia-0.5.2\bin
julia.exe --worker IFiJUqV4N1MuC9Uo | telnet windows-master 8449

```

But after the execution in stderr I can see

```julia
ERROR: write: broken pipe (EPIPE)
 in yieldto(::Task, ::ANY) at .\event.jl:136
 in wait() at .\event.jl:169
 in stream_wait(::Task) at .\stream.jl:44
 in uv_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at .\stream.jl:820
 in unsafe_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at .\stream.jl:830
 in write(::Base.PipeEndpoint, ::Array{UInt8,1}) at .\io.jl:175
 in print at .\strings\io.jl:70 [inlined]
 in start_worker(::Base.PipeEndpoint, ::String) at .\multi.jl:1539
 in process_options(::Base.JLOptions) at .\client.jl:218
 in _start() at .\client.jl:321

```

Have you encountered smth similar? Do you have any ideas how to fix it?

---

<div class="post-metadata">

**Author:** ![John\_Hearns](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/john_hearns/32/1685_2.png) [@John\_Hearns](https://discourse.julialang.org/u/John_Hearns)\
**Post date:** [June 22, 2017, 8:09am UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/8 "2017-06-22T08:09:32Z")

</div>

Ivborrisov, you have to ask yourself what is the cost/benefit analysis here?  
If you have to expend a lot of effort to get it running on Windows then is it worth the effort.  
By that I mean:  
how much would it be in terms of money cost or effort to get Linux servers  
Or hire them on a cloud provider

Being more constructive - you have Windows servers. Run Linux on VMs on those windows servers… set up your cluster!

---

<div class="post-metadata">

**Author:** ![ivborissov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivborissov/32/2841_2.png) [@ivborissov](https://discourse.julialang.org/u/ivborissov)\
**Post date:** [June 22, 2017, 1:09pm UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/9 "2017-06-22T13:09:27Z")

</div>

John, thanks!  
Finally I am using ClusterManagers/htcondor with my windows servers. I had to customize condor.jl to support Win environment but now it is working

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 22, 2017, 2:05pm UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/10 "2017-06-22T14:05:45Z")

</div>

Could you share what you did? This is the kind of thing that needs to be done once by someone very motivated. Glad to hear it worked out 🙂

---

<div class="post-metadata">

**Author:** ![ivborissov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivborissov/32/2841_2.png) [@ivborissov](https://discourse.julialang.org/u/ivborissov)\
**Post date:** [June 22, 2017, 2:50pm UTC](https://discourse.julialang.org/t/building-cluster-for-julia-parallel-computations/3848/11 "2017-06-22T14:50:37Z")

</div>

Sure,  
I have put my version of condor.jl here

> <https://github.com/ivborissov/ClusterManagers.jl/tree/ivborissov/condor_win/src>
>
> //github.com/ivborissov/ClusterManagers.jl/tree/ivborissov/condor\_win/src

I have to make it more “prod-ready” because now it is more like a draft but it is working (on 0.5.2)  
Please, see requirements in README  
I tried to use default telnet to send ip and port details from remote workers but had BROKEN PIPE messages so didn’t have much time to investigate and I used ncat on remote workers:

> **[Ncat - Netcat for the 21st Century](https://nmap.org/ncat/)**
>
> Ncat is a free, open-source Netcat replacement for Linux, Windows, OS X and more. TLS/SSL encryption, proxy support, IPv6, Lua scripting.

For the multithread case on remote workers you can uncomment the line  
println(scriptf, “set JULIA\_NUM\_THREADS=%NUMBER\_OF\_PROCESSORS%”)