What's the easiest way to set up multi-node distributed computing with Julia?

New to Julia
1

I was thinking, is it easier to just buy another Raspberry pi, install Julia and then try to set up SSH into it?

Or is it just to spinning a few EC2 instances and do it from there?

Does anyone have advice on what can be done with setting a distribute environment that is actually multinode not just multiprocess on the same machine?

Ideally, it will be easy to maintain and cheap.

2

You should only go the RPi route if you actually want to tinker with the hardware. I have not used mine in a while, but I know it would be silly to try to do any real work on it. They just aren’t as powerful as you’d hope. Even the smallest EC2 would beat it. Perhaps it would be a lower cost way to browse the Discourse…

2 Likes
3

AWS Parallelcluster is easy to set up. You can set it up on the free tier to play around with.
As usual though - make sure to terminate all instances and gateways which you are not using.

Should you wish to go with Azure I can get you some advice.

1 Like
4

Coming soon:

Doesn’t get much easier than that!

I like playing with hardware.
Right now, I have:

  • 10980XE, overclocked to 4.1 GHz AVX512 all-core (18 cores)
  • 7980XE, overclocked to 3.8 GHz AVX512 all-core (18 cores)
  • 7900X, stock speed I think, 3.3 GHz AVX512 all-core (10 cores)

I also have a 9940X (14 cores) sitting in a box, and an old infiniband switch.
I plan on getting around to experimenting with running it as a cluster at some point, so I’d be interested in suggestions on the easiest way. Probably SSHManager, although the different clock speeds probably means I’d need pmap to distribute work?

2 Likes
5

Cluster toolkits:
https://openhpc.community/

Bright Cluster Manager - free for 8 nodes Easy8

Qlustar https://qlustar.com/

If anyone is building an HPC cluster please PM me. That what I do for a day job.

4 Likes