Not to pull this thread too far off-topic, but for a while I’ve been contemplating writing a Julia equivalent of Clojure’s nREPL (networkedREPL). The idea is to split a REPL into front-end (handling code-entry, history, completion, display, etc.) and a back-end (handling evaluation and execution) so that multiple different front-ends (console, IDE, other) can communicate with the same back-end.
I know the Julia REPL already has some network abilities built in, but IIRC the distinction between front- and back-end is not quite so clear cut. Would there be interest in something like this?
@jballanc: I was under the impression that this is what the Juno code and VS code extension language server do. I haven’t looked into the details, but my only concern is that it would be ideal if there was
a standardized, IDE-agnostic backend,
which various IDE protocols could build on.
I was tempted to roll my own, but I really think that one-person projects are not the way to go. OTOH, rolling my own Emacs frontent for some custom protocol is not ideal either. So there is a delicate balance here.
The REPL design is fairly separable – Keno has done REPL demos using local-remote sessions (but IIRC some of that code was ripped out before merging to base). It “just” needs to be hooked up again
Also, IJulia supports the Jupyter server protocol, which has years of design iteration behind it (for better and worse). So one approach would be to teach the REPL front end to use that.
(I would use the Julia REPL for Python too, if that were available, because it is dramatically better than even the latest IPython iterations based on python-prompt-toolkit)
My thoughts exactly. In Clojure’s case, the wire protocol is BEP (the BitTorrent wire protocol), and the application protocol has messages encoded as maps. The advantage to this is that it is possible to build in middleware that modifies messages to/from the REPL, extending the interface fairly easily.
Yeah, last time I (briefly) looked into this I noticed that Base.REPL implements a StreamREPL type (here), including an intriguingly named start_repl_server function…that isn’t used anywhere (that I could find).
I agree that this is definitely worth looking into. No need to re-invent the wheel. Without having peeked under the covers of IJulia, my instinctive first reaction/concern is that the REPL backend should be as simple and lightweight as possible. i.e. No extra machinery for managing processes. If the IJulia server code can be split out to isolate just the server communication, I think that should work.
Having worked with many Ruby and Clojure REPLs, my personal list of wants would be:
Transport-independent wire protocol (preferably based on prior art). Sending objects over TCP sockets might be convenient, but is too fragile.
Documented, backwards-/forwards-compatible application protocol. I shouldn’t have to downgrade my local install of Julia just to REPL-connect to a server running a prior version. Similarly, IDEs should be able to connect to Julia REPLs from various different Julia versions without requiring version shims.
Extensible application protocol to facilitate the development of middleware.
Server should be embeddable into existing applications.
IJulia already is just the backend communication. All of the front end, kernel launching, etcetera, is managed by Jupyter. If you want to replace Jupyter with your own front-end code, IJulia won’t care as long as you talk to it with the same ZMQ protocol.
IJulia/Jupyter uses ZeroMQ, which can go over inproc, IPC, TCP, TIPC, and multicast.
Jupyter already does this. Not only does it talk to different versions of Julia, it talks to different languages!
Not sure what you mean, precisely, but the Jupyter protocol is pretty extensible. You can add new message types and new data to existing messages without disrupting existing backends.
Not sure which you’re calling the “server” here, but IJulia could easily be modified to be embeddable.
Ok, just because I don’t have anywhere else public to make notes, this is just a quick rundown of potential issues I see with adopting IJulia as a REPL server. I am by no means an expert, and this is quite literally the first look I’ve had at the IJulia code, so please feel free to correct mistakes if I’ve made any, and I consider all points open for debate!
Pros
Transport is ZeroMQ. In an ideal world, transport would be pluggable. In the real world, ZeroMQ is as close to ideal as one can get. (In case it’s not clear, I’m a huge fan of ZeroMQ.)
Two users connected to the same REPL server shouldn’t have their histories interleaved
Keeping history client-local is useful for replaying commands across servers. e.g. Many times with Clojure I’ve connected to a staging REPL, evaluated some code then, after confirming the code does what I want, reconnect to a production REPL and re-eval the same code from my local history.
That said, having history on the server does not preclude client history, though managing two histories could be confusing.
Overall, I think IJulia could definitely form a solid base for a Julia REPL server. What I would suggest, on a high level, is the following:
Modularize. Off the top of my head:
Core
IO
MessageProtocol (decode & encode for client & server)
Evaluation
Middleware
ModeSwitch
History
InlineDisplay
Make ModeSwitch pluggable, provide default plugins for Help and Shell
Implement IJulia server as Core + all Middleware
Implement JuliaREPLServer as just Core + ModeSwitch
So far the biggest potential hurdle I see is that the Jupyter message protocol is too corse-grained. For example, it would be nice to be able to implement code completion as a server-side plugin, since the server will have full context for what symbols are in scope. However, this would necessitate the ability to send “action” events (e.g. user pressed <TAB> requesting a completion). Another example is mode switching. The server should contain information about what modes are available, but switching between modes (e.g. changing the REPL prompt to help> when the ? key is pressed) would require messages to be sent on each key press. (An alternative implementation would have the ModeSwitch middleware implemented both client and server-side, but this would still require some means of communicating non-code messages client-to-server, e.g. to establish which ModeSwitches were currently installed.)
That’s all I have time to write up for the moment. If this seems like a good start on a rough outline, I’d be happy to turn some of these points into GitHub tickets (just wanted to be able to have some discussion around the issue first).
No, they can’t be. The client (e.g. the Jupyter notebook) knows nothing about Julia, and has no access to Julia objects. Only the kernel (server) can take a Julia object and call methods on it in order to create text/plain or image/png or whatever representation to send to the server.
New message types is definitely part of it (and good to see that it’s supported). The other part is being able to intercept/modify/re-route messages coming and going (think HTTP). Having library support for all of this (esp. encode/modify/decode) is a definite bonus.
Essentially, the server is the “thing doing the evaluating” (or in the case of shell/help mode, the thing doing the shelling-out/doc lookup). In Clojure, I’ve very frequently embedded an nREPL server in my deployed applications, having it listen on a local port. Then, with an nREPL client and an SSH tunnel, I can REPL-in to running applications. This is tremendously useful for debugging (and even, if you have the guts for it, live patching).
Ok, that’s what I suspected. When I said they should be “client concerns”, what I meant is that ideally the server only concerns itself with evaluation, not representation. i.e. A graphical IDE may have a very different idea of how to represent a Julia object returned from an evaluation than a textual console.
In thinking about this, I was drawn between two design concepts.
the Jupyter “server” (i.e. current IJulia project) is itself a “client” of the REPL server (i.e. Jupyter client → IJulia front-end/REPL-client → Julia REPL server)
introduce the concept of server middleware
Solution #2 might require a bit more conceptual design (when is middleware setup/configured in the life of the REPL server stack? can one REPL “core” server multiple server middleware stacks?), but in the end I think it will be simpler overall (hence why I suggested it above).
Edit: Actually, the more I think about this, the more I think the issue of how to display objects should be handled as a (potentially middleware-driven) client-server negotiation. I wouldn’t want to have to spin up separate REPLs just because I want to connect on the command-line and in an IDE at the same time, but I can definitely see that the IDE would want to be able to display objects with legitimate image representations as images.
The way the Jupyter protocol works is that the server sends out several representations of an object, and the client(s) decide how to display it. It must always send a text/plain representation, but may also send e.g. img/png or text/html. Usually, non-image representations will be small, so there is no problem in sending several textual representations along with perhaps one image representation.
Hmm…I would worry that image, or even say video, representations could get large and cause network issues. Would still be nice for clients to have a way to indicate that there are representations they would get no use from, so there is no need to send them. But I’ll concede that this is a minor point and that it looks like this aspect of the Jupyter message protocol should work for a REPL.
After thinking about it a bit, I think the best place to start would be to add some more tooling around the message protocol. Once that’s available, then rearranging the other pieces and extracting what needs extracting to make IJulia a good, general purpose REPL backend will be significantly easier.
Would such tooling be better as a separate project? Or should it remain within IJulia? (I can see pros/cons either way…)
It can handle scoped eval, autocomplete and docs like Juno does (the addition of Base.include_callbacks in Julia 0.7 makes this way easier), but the emphasis is very much on the word ‘hacked’
The julia-repl emacs package works pretty well but it doesn’t have a backchannel to reply to the editor for autocomplete etc. Same for ESS. LanguageServer.jl doesn’t seem to support eval and as far as I can tell the autocomplete doesn’t take into account the local scope. Same for IJulia.
I would prefer to use a standard backend if one has been agreed on. Not so much a standard network protocol, but maybe a CodeTools 2.0 that contains tricky things like how to correctly redefine a module which may or may not be a root module, or how to figure out what module is in scope at a particular point in a file. REPL would be a good place for these, even.
I’ll start by making a PR this weekend for REPLCompletions that allows setting the module and returns structured completions instead of bare strings. I’d appreciate feedback on where to go from there. Also if anyone would care to review/sanity-check my module loading code that would be a great comfort - https://github.com/jamii/exo/blob/master/Emacs.jl#L35-L92
I also have similar stuff for GtkIDE, although it’s Julia to Julia communication and it’s not very good. One thing that I need is the ability for the eval part to return a Julia object (like a plot) and serialize it back to the client. Note that the evaluation also need to be interruptible.
@Tamas_Papp Btw this works ok alongside julia-repl. I run Emacs.start_server() from julia-repl and do eval with my code, but still get the clickable file locations from julia-repl. I haven’t figured out a way to do those outside of emacs yet.
Another thing I’m stuck on is inserting text into the repl itself, rather than faking a repl prompt. That way stuff that’s evaled from Emacs would still be in the repl history, and prompt-pasting would work without me need to re-implement it. I feel like it should be possible to just insert text into the lineedit input somewhere, but everything I’ve tried so far has produced text in the terminal but not been recognized by the repl itself.
Yeah, that’s another reason to want to reuse the existing repl infrastructure. I wonder if it’s possible to get the StreamREPL in the stdlib working again? Then the frontend just needs to handle networking and figuring out what module we’re in.
Great to hear! Feel free to ping me if I can help with a code review or something. We really need this for Juno as well (plus nice async printing, which I hope to look into before 0.7 is out).
