How well does Julia support distributed processing?

[stefkuypers]

Hi,

We’re looking at using Julia for large scale simulations and are wondering how well Julia supports distributed processing. Where do we need to look for libraries supporting this? Are there any example projects we could look at?

Thanks in advance,
Stef Kuypers

[Sukera]

There’s the Distributed stdlib, on top of which ClusterManagers.jl is built. There’s also “regular” MPI if you want to do the process management manually (though I prefer ClusterManagers). Transducers.jl is also worth mentioning here.

More specific advise will depend on what kind of processing you’re doing, what your environment looks like etc. “Distributed Computing” is a kind of vague term, so do you have a certain application/usecase in mind?

[stefkuypers]

Thanks!

Yes, the use case is economic simulations. I’m starting to work together with Jacky Mallett who created the ThreadNeedle framework and we’re looking into the possibility to do something similar in Julia.

[Sukera]

https://github.com/jackymallett/Threadneedle for other people looking through this thread, this seems to be the repo?

If the use case is financial modeling, I think you’re already kind of behind the curve. Looking through the case studies on juliacomputing.com:

• Solvency II Compliance
  • One of Europe’s largest insurers is using Julia for Solvency II compliance
• Financial Modeling at Atteson Research
• Now-Casting Economics uses Julia to reduce macroeconomic modeling time from weeks to days
• Conning is using Julia in large scale Monte Carlo simulations for insurance risk assessment solutions
  • They’re explicitly saying they chose julia for distributed parallel computing capabilities
• The New York Federal Reserve also uses julia, they’ve put their code on Github as DSGE.jl and they showed a considerable performance increase (those numbers are from julia 0.4.5, but as julia has only gotten faster, I think those speedups have also gone up - it would be interesting to know about how fast their modeling is nowadays).
• Thomas J. Sargent also held an opening keynote for juliacon 2016, titled “Quantitative Macroeconomics in Julia”.

So from what I can tell, there are already quite a few actors in that space using julia for that purpose, quite a few of which are also using this forum or are on julialang.zulipchat.com. I sadly don’t know these people by name, as that is not my area of expertise, but I’ve linked this thread on zulip so maybe people with that expertise can chime in.

[stefkuypers]

Yes, that’s the repo.

Thanks a lot! We’ll look into it.

[Sukera]

My best advise would be to just try things out and if any specific questions arise, you can ask them here on discourse, on the julialang zulip or on slack. We’re always happy to help

[stefkuypers]

Thanks!