# How to submit distributed memory jobs to cluster?

**URL:** <https://discourse.julialang.org/t/how-to-submit-distributed-memory-jobs-to-cluster/63388>\
**Category:** General Usage\
**Tags:** distributed\
**Created:** [June 22, 2021, 5:32pm UTC](https://discourse.julialang.org/t/how-to-submit-distributed-memory-jobs-to-cluster/63388 "2021-06-22T17:32:37Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![biona001](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/biona001/32/16497_2.png) [@biona001](https://discourse.julialang.org/u/biona001)\
**Post date:** [June 22, 2021, 5:32pm UTC](https://discourse.julialang.org/t/how-to-submit-distributed-memory-jobs-to-cluster/63388/1 "2021-06-22T17:32:38Z")

</div>

I have a `run_jobs.jl` file that supports distributed memory parallelism. How should the beginning of `run_jobs.jl` look like if I want to run it on a cluster (univa grid engine) via `qsub`?

I can use [ClusterManagers.jl](https://github.com/JuliaParallel/ClusterManagers.jl) to run my code _interactively_ by

```julia
# addprocs_sge will request 16 cores (possibly on different nodes)
using ClusterManagers, Distributed
ClusterManagers.addprocs_sge(16; qsub_flags=`-l h_rt=24:00:00,h_data=4G,arch=intel-gold-61\*`)

# run my actual code that supports distributed memory
...

```

But this job will run for a long time, and I don’t want to wait until it finishes.

If I `qsub` a single-core job to run the script above, I get this error:

```nohighlight
Base.IOError("could not spawn `qsub -N julia-2229 -wd /u/home/b/biona001 -terse -j y -R y -t 1-4 -V -l 'h_rt=24:00:00,h_data=4G,arch=intel-gold-61*'`: no such file or directory (ENOENT)", -2)`

```

I also tried to submit a distributed memory job (specify `-pe dc* 16` in my shell script), and in `run_jobs.jl` the heading is simply `using Distributed; addprocs(16)`, the job will crash and cause core dumps. If I change the job submission to shared memory node (specify `-pe shared 16`), the job will run successfully on a single node with 16 cores, but this will massively increase queue time.

Any tip is appreciated. Thanks!