# Julia parallel computing over multiple nodes in SGE cluster

**URL:** <https://discourse.julialang.org/t/julia-parallel-computing-over-multiple-nodes-in-sge-cluster/3152>\
**Category:** New to Julia\
**Tags:** parallel\
**Created:** [April 10, 2017, 10:24pm UTC](https://discourse.julialang.org/t/julia-parallel-computing-over-multiple-nodes-in-sge-cluster/3152 "2017-04-10T22:24:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![oskar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oskar/32/12494_2.png) [@oskar](https://discourse.julialang.org/u/oskar)\
**Post date:** [April 10, 2017, 10:24pm UTC](https://discourse.julialang.org/t/julia-parallel-computing-over-multiple-nodes-in-sge-cluster/3152/1 "2017-04-10T22:24:12Z")

</div>

I actually already asked this on StackOverflow but did not find any answer that solved my problem.

Here is the link to the question:

> <https://stackoverflow.com/questions/43079309/julia-parallel-computing-over-multiple-nodes-in-cluster>

I basically want to run Julia on multiple nodes on an SGE cluster with the opportunity to use the native `@parallel` parallelization macro. I tried with `julia --machinefile $PE_HOSTFILE` and also using the `ClusterManagers.addprocs_sge(X)` (X == number of procs) but they both failed.

Can anyone give me any advice?

---

<div class="post-metadata">

**Author:** ![bjarthur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bjarthur/32/9638_2.png) [@bjarthur](https://discourse.julialang.org/u/bjarthur)\
**Post date:** [April 11, 2017, 3:31pm UTC](https://discourse.julialang.org/t/julia-parallel-computing-over-multiple-nodes-in-sge-cluster/3152/2 "2017-04-11T15:31:40Z")

</div>

try `addprocs_qrsh` instead. if your file system is tuned for high throughput and the cost of high latency, inter-process communication through the filesystem can timeout.

---

<div class="post-metadata">

**Author:** ![oskar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oskar/32/12494_2.png) [@oskar](https://discourse.julialang.org/u/oskar)\
**Post date:** [April 11, 2017, 8:57pm UTC](https://discourse.julialang.org/t/julia-parallel-computing-over-multiple-nodes-in-sge-cluster/3152/3 "2017-04-11T20:57:04Z")

</div>

Ok I was told by the IT team that this methods need passwordless ssh, whereas the cluster I’m using doesn’t allow that. They’ll look into this in the following days, I’ll post as soon as I have a solution if I find one.

---

<div class="post-metadata">

**Author:** ![oskar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oskar/32/12494_2.png) [@oskar](https://discourse.julialang.org/u/oskar)\
**Post date:** [April 16, 2017, 4:52am UTC](https://discourse.julialang.org/t/julia-parallel-computing-over-multiple-nodes-in-sge-cluster/3152/4 "2017-04-16T04:52:38Z")

</div>

The IT team got back to me and told me that the SGE does not allow passwordless ssh, that’s why addprocs\_sge() or any other way I tried wouldn’t work. However they now added a file for the job that I can pass to Julia and told me to run the job with this script:

```julia
qlogin -pe mpi_28_tasks_per_node 56
module load julia/0.5.1
julia --machinefile $TMPDIR/machines

```

The machines file looks like this:

```julia

::::::::::::::
/scratch/8548498.1.u/machines
::::::::::::::
{hostname1}
{hostname1}
...
{hostname2}
{hostname2}

```