Hi,
In my workflow I need to run several simulations which make use of the DifferentialEquations package.
To automate running the simulations I need to start the simulations from command line (and not within the Julia REPL):
julia sim1.jl
julia sim2.jl
julia sim3.jl
etc…
The problem I’m facing is that the startup time is large and it constitutes the major part of running “julia simX.jl”. Using PackageCompiler to shorten startup time is not an option because I cannot make it work reliably on all the workstations I have to use to run the tests.
After some thinking I started wondering if it is possible to connect to an existing REPL and run the tests sequentially from my remote connection. I looked at the documentation of the Distributed package and it seems like there is all the stuff I need but I couldn’t figure out how to get what I wanted. What I’d like to achieve is the following (example for 2 simulations)
connect to the REPL
run the simulation 1
disconnect from the REPL. The REPL continues to exist.
connect to the same REPL
run the simulation 2
disconnect from the REPL.
Is it possible to attach the current julia process to an existing REPL (remote or on the same computer)?
If it is not possible, what can I do to speed up my work?
Carlo, this is NOT a direct answer to your question. An expert will come along in a minute.
If you use an ssh connection to a remote system you can use a utility called ‘screen’
A screen session will keep an application running and you can disconnect and reconnect to the terminal.
Have you considered using a jupyter notebook?
That should allow using a remote julia session (which stays alive)
Or is this not an option for you?
You can check this out on juliabox
Apologies if I misunderstood your problem.
@johnh and @pfitzseb
I knew about screen but it didn’t came into my mind to solve the problem. I think with screen I could get what I want but if I could attach and detach to a REPL using the same tools used for distributed computing I could also get the results of my simulations which could help me understand if everything went ok or not. Probably I can do it also with screen but it would be more complicated.
Attaching a REPL to a running remote process is in principle possible but no one has done the work to make it happen. Feel free to open a feature request issue on GitHub. Note that there are pretty hefty security implications of allowing such a thing, so it’s not just a matter of making it work, but also of making it work safely.
Thank you for the feedback. Would it be a quite unsafe thing also using the Distributed.jl package infrastructure? I don’t see a big difference between attaching to a running Julia process and distributed computing with Julia.
For server-side long-running applications (everything web dev) this would be a very valuable feature. For reference, it is available on Erlang/Elixir. The value resides in the fact that
a. one can attach to a running session to debug issues and inspect application state
b. we could theoretically update the running code by manually triggering Revise to inject application updates without restarting the app (no downtime)
If you start remote IJulia via remote_ikernel you can attach as many local clients as you want. The catch is that you have to start IJulia via remote_ikernel and loosing connection will kill the remote process. I’ve been wanting to write a “Jupyter kernel proxy” that can bridge the pre-existing remote Jupyter kernel and local client though SSH. This would help not only Julia community but also entire scientific computing community.
+1 for tmux, I’ve also used it with Julia apps started from the REPL. The downside is tmux has a pretty high learning curve (and I still can’t remember the shortcuts, I’m having all these tmux cheat-sheets around). Will give screen a try, seems more user friendly.
I use basically 3-4 of them, for creating a new terminal, stepping back and forth between them, and killing an unresponsive one.
No doubt there must be a lot of excellent features I am missing with this approach, but I don’t spend enough time in a terminal to make learning all of them worthwhile.
A quick follow up based on my current struggles at work:
1 - tmux / screen works in some cases, but in production, most certainly, we need the web apps demonized so that they’re automatically restarted upon crashing. This means that the Julia process will be started by something like supervisord not by the user, so we need to attach to that process.
2 - found byobu which is a more user friendly wrapper around tmux and screen
How about launching a IJulia kernel and then use @spwan/@async to start your webapp inside of it? You can then attach a REPL to it using jupyter console etc.
I don’t think you need to know IJulia internal to use it. There is no direct API to do it but I think you can put something like @spawn start_my_app() in ~/.julia/config/startup_ijulia.jl. Yeah, I know it’s a bit of a dirty trick but maybe a good start for trying things out. Guarding it with some environment variable (which is set in supervisord configuration) probably is a good idea. So, the invocation you need to use (e.g., put in supervisord configuration) would be something like START_MY_APP=yes jupyter console --kernel=julia-1.x. To attach to it, use jupyter console --existing KERNEL_ID. See: Jupyter console 6.0 — Jupyter console 6.0.0dev documentation
Sorry, I’ve never tried this (IJulia + web app) myself so don’t know. I know that a Jupyter kernel needs to open some ZMQ connections (which uses a bunch of TCP ports) to talk with Jupyter frontends. Maybe this can interfere with I/O performance of your web app? For CPU-bound task, my guess is that it won’t matter since people do computations in Jupyter notebooks and they’d complain if there is any performance disadvantage.
