Connecting RShiny to a running instance of Julia

General Usage
1

I have a simulation model in Julia that has the following workflow

using SimModel
addprocs(SlurmManager(512))  
@everywhere using SimModel
run_simulation(params) # function uses pmap to launch work() over 500 processors

Something important to note is that the statement @everywhere using SimModel takes about 7 minutes to run. The actual run_simulation is about 20 seconds.

What I would like to do is have a web facing interface for my model. I am mostly experienced with RShiny, so that’s what I am considering but alternatives are welcome.

My idea is to have a user select the parameters using RShiny app, which gets saved as a JSON file, and when user clicks “run”, the julia script executes, reads the JSON file for its parameters, and executes the model. Something like julia myscript.jl where myscript.jl would run those commands above.

The problem here is obvious, which is that running julia myscript.jl will take about 7 minutes to set up the parallel workers as well compile all the functions over and over again.

Ideally what I would like to do have a Julia instance running where I’ve already allocated the nodes (i.e. addprocs and the script is basically just “waiting” for input. As soon as user submits their parameters using RShiny, all i have to do is run run_simulation(params).

Any one have an idea on how this can be accomplished?

2

You can use PackageCompiler.jl to compile the package before putting it online.

3

I think the problem still remains that connecting to the workload manager takes about 7-8 minutes, and I don’t think packagecompiler will help with that, but i will try.

4

Oh I see. I was thinking your issue was compile time, sorry. For that issue, the problem is likely that Julia defaults to an all-to-all topology, but you don’t necessarily need that. Changing the topology can help quite a bit. Doing something like addprocs(SlurmManager(512),topology=:master_worker) might give you all of the connections you need, which is exponentially less than all-to-all.

5

That’s exactly what I did in my code (see the previous thread), but its still about 7-8 minutes… I will actually try to 1.4 to see if things have improved.