# Using Distributed Computing with AWS Batch/ AWS EC2

**URL:** <https://discourse.julialang.org/t/using-distributed-computing-with-aws-batch-aws-ec2/51815>\
**Category:** General Usage\
**Tags:** parallel\
**Created:** [December 14, 2020, 5:53pm UTC](https://discourse.julialang.org/t/using-distributed-computing-with-aws-batch-aws-ec2/51815 "2020-12-14T17:53:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![bmalham](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bmalham](https://discourse.julialang.org/u/bmalham)\
**Post date:** [December 14, 2020, 5:53pm UTC](https://discourse.julialang.org/t/using-distributed-computing-with-aws-batch-aws-ec2/51815/1 "2020-12-14T17:53:57Z")

</div>

I am currently having issues using parallel processing using Julia’s Distributed library on an EC2 instance. My script attempts to speed up a series of sequential actions by putting some parts in parallel using the _remotecall_ function. The code in question that I put in parallel copies a large number of files into a temp directory and performs some operations on them. Running this on my local machine via docker gives me the expected result and shows an expected speedup in terms of runtime. When I upload the exact same docker images to AWS ECR and start a Batch job which runs the same code in the same docker image that was successful locally on EC2. When running on EC2 the parallel steps appear to run correctly but later down the line the information that was generated during those steps isn’t available. Does anyone have experience using Julia’s Distributed library with EC2 instances or just with virtual CPUs in general, and did you have to do anything differently than with normal CPUs? Thanks in advance for any help.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [December 14, 2020, 6:46pm UTC](https://discourse.julialang.org/t/using-distributed-computing-with-aws-batch-aws-ec2/51815/2 "2020-12-14T18:46:58Z")

</div>

Very stupid reply - the temp directory is going to be local to the individual VM. SO it will be different for each VM.

Are you using AWS pcluster to set up the cluster? You can request a filesystem which is common to all the VMs. Please forgive me if I do not understand the problem!  
Duuh - read the post John. Its docker on ECR

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [December 14, 2020, 6:52pm UTC](https://discourse.julialang.org/t/using-distributed-computing-with-aws-batch-aws-ec2/51815/3 "2020-12-14T18:52:26Z")

</div>

> **[Amazon ECS and AWS Fargate support for Amazon EFS File Systems now generally...](https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-ecs-aws-fargate-support-amazon-efs-filesystems-generally-available/)**

If you use pcluster for a more traditional HPC type setup you can have Luster FsX storage

> **[Building an HPC cluster with AWS ParallelCluster and Amazon FSx for Lustre |...](https://aws.amazon.com/blogs/storage/building-an-hpc-cluster-with-aws-parallelcluster-and-amazon-fsx-for-lustre/)**
>
> High performance computing (HPC) customers love the breadth of services offered by AWS and the flexibility offered by the cloud to address their computational challenges. AWS provides you with the opportunity to innovate quickly and accelerate your...

---

<div class="post-metadata">

**Author:** ![bmalham](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bmalham](https://discourse.julialang.org/u/bmalham)\
**Post date:** [December 14, 2020, 10:29pm UTC](https://discourse.julialang.org/t/using-distributed-computing-with-aws-batch-aws-ec2/51815/4 "2020-12-14T22:29:50Z")

</div>

Thanks for the reply!

Maybe my post was a bit misleading, I’m just using the single EC2 instance. I guess the library itself ‘Distributed’ is a bit misleading as I’m only trying to use one computer, but my intention was to take advantage of the multiple vCPUs provided by EC2.