How well does Julia support distributed processing?

Hi,

We’re looking at using Julia for large scale simulations and are wondering how well Julia supports distributed processing. Where do we need to look for libraries supporting this? Are there any example projects we could look at?

Thanks in advance,
Stef Kuypers

There’s the Distributed stdlib, on top of which ClusterManagers.jl is built. There’s also “regular” MPI if you want to do the process management manually (though I prefer ClusterManagers). Transducers.jl is also worth mentioning here.

More specific advise will depend on what kind of processing you’re doing, what your environment looks like etc. “Distributed Computing” is a kind of vague term, so do you have a certain application/usecase in mind?

2 Likes

Thanks!

Yes, the use case is economic simulations. I’m starting to work together with Jacky Mallett who created the ThreadNeedle framework and we’re looking into the possibility to do something similar in Julia.

GitHub - jackymallett/Threadneedle: Economic and Banking System Simulation Framework. for other people looking through this thread, this seems to be the repo?

If the use case is financial modeling, I think you’re already kind of behind the curve. Looking through the case studies on juliacomputing.com:

So from what I can tell, there are already quite a few actors in that space using julia for that purpose, quite a few of which are also using this forum or are on julialang.zulipchat.com. I sadly don’t know these people by name, as that is not my area of expertise, but I’ve linked this thread on zulip so maybe people with that expertise can chime in.

1 Like

Yes, that’s the repo.

Thanks a lot! We’ll look into it.

My best advise would be to just try things out and if any specific questions arise, you can ask them here on discourse, on the julialang zulip or on slack. We’re always happy to help :slight_smile:

2 Likes

Thanks!