New message types is definitely part of it (and good to see that it’s supported). The other part is being able to intercept/modify/re-route messages coming and going (think HTTP). Having library support for all of this (esp. encode/modify/decode) is a definite bonus.
Essentially, the server is the “thing doing the evaluating” (or in the case of shell/help mode, the thing doing the shelling-out/doc lookup). In Clojure, I’ve very frequently embedded an nREPL server in my deployed applications, having it listen on a local port. Then, with an nREPL client and an SSH tunnel, I can REPL-in to running applications. This is tremendously useful for debugging (and even, if you have the guts for it, live patching).
Ok, that’s what I suspected. When I said they should be “client concerns”, what I meant is that ideally the server only concerns itself with evaluation, not representation. i.e. A graphical IDE may have a very different idea of how to represent a Julia object returned from an evaluation than a textual console.
In thinking about this, I was drawn between two design concepts.
the Jupyter “server” (i.e. current IJulia project) is itself a “client” of the REPL server (i.e. Jupyter client --> IJulia front-end/REPL-client --> Julia REPL server)
introduce the concept of server middleware
Solution #2 might require a bit more conceptual design (when is middleware setup/configured in the life of the REPL server stack? can one REPL “core” server multiple server middleware stacks?), but in the end I think it will be simpler overall (hence why I suggested it above).
Edit: Actually, the more I think about this, the more I think the issue of how to display objects should be handled as a (potentially middleware-driven) client-server negotiation. I wouldn’t want to have to spin up separate REPLs just because I want to connect on the command-line and in an IDE at the same time, but I can definitely see that the IDE would want to be able to display objects with legitimate image representations as images.
The way the Jupyter protocol works is that the server sends out several representations of an object, and the client(s) decide how to display it. It must always send a text/plain representation, but may also send e.g. img/png or text/html. Usually, non-image representations will be small, so there is no problem in sending several textual representations along with perhaps one image representation.
Hmm…I would worry that image, or even say video, representations could get large and cause network issues. Would still be nice for clients to have a way to indicate that there are representations they would get no use from, so there is no need to send them. But I’ll concede that this is a minor point and that it looks like this aspect of the Jupyter message protocol should work for a REPL.
After thinking about it a bit, I think the best place to start would be to add some more tooling around the message protocol. Once that’s available, then rearranging the other pieces and extracting what needs extracting to make IJulia a good, general purpose REPL backend will be significantly easier.
Would such tooling be better as a separate project? Or should it remain within IJulia? (I can see pros/cons either way…)
It can handle scoped eval, autocomplete and docs like Juno does (the addition of Base.include_callbacks in Julia 0.7 makes this way easier), but the emphasis is very much on the word ‘hacked’
The julia-repl emacs package works pretty well but it doesn’t have a backchannel to reply to the editor for autocomplete etc. Same for ESS. LanguageServer.jl doesn’t seem to support eval and as far as I can tell the autocomplete doesn’t take into account the local scope. Same for IJulia.
I would prefer to use a standard backend if one has been agreed on. Not so much a standard network protocol, but maybe a CodeTools 2.0 that contains tricky things like how to correctly redefine a module which may or may not be a root module, or how to figure out what module is in scope at a particular point in a file. REPL would be a good place for these, even.
I’ll start by making a PR this weekend for REPLCompletions that allows setting the module and returns structured completions instead of bare strings. I’d appreciate feedback on where to go from there. Also if anyone would care to review/sanity-check my module loading code that would be a great comfort - https://github.com/jamii/exo/blob/master/Emacs.jl#L35-L92
I also have similar stuff for GtkIDE, although it’s Julia to Julia communication and it’s not very good. One thing that I need is the ability for the eval part to return a Julia object (like a plot) and serialize it back to the client. Note that the evaluation also need to be interruptible.
@Tamas_Papp Btw this works ok alongside julia-repl. I run Emacs.start_server() from julia-repl and do eval with my code, but still get the clickable file locations from julia-repl. I haven’t figured out a way to do those outside of emacs yet.
Another thing I’m stuck on is inserting text into the repl itself, rather than faking a repl prompt. That way stuff that’s evaled from Emacs would still be in the repl history, and prompt-pasting would work without me need to re-implement it. I feel like it should be possible to just insert text into the lineedit input somewhere, but everything I’ve tried so far has produced text in the terminal but not been recognized by the repl itself.
Yeah, that’s another reason to want to reuse the existing repl infrastructure. I wonder if it’s possible to get the StreamREPL in the stdlib working again? Then the frontend just needs to handle networking and figuring out what module we’re in.
The original motivation for julia-repl was to punt to Julia’s REPL implementation for all of the extra features (documentation, Pkg, shell) by pretending to be a terminal. Unfortunately, that design has proven to be a dead end, because Emacs’s terminal support is buggy.
A major reconsideration of this approach is necessary, which will probably involve implementing an interface for quite a bit of functionality. While this is a burden, at the same time it can probably be made more friendly (eg make docstrings appear in a new window, render them from Markdown or HTML, add a menu-driven interface for Pkg3 functionality, etc) — my ideal is something like SLIME. I will probably also involve some PRs to packages and Base separating functionality and interface.
Jumping to source code locations should be trivial, just filtering for a regex. I will look into making it work with your code.
A major reconsideration of this approach is necessary, which will probably involve implementing an interface for quite a bit of functionality.
That’s the approach followed by ESS, which works pretty well. If you’re considering moving away from the terminal approach (which I think is a good idea, terminal support in emacs is crap), please give it a go: it would be very nice to reduce the fragmentation of the emacs-julia ecosystem.
We have similar issues with the new Juno REPL (not quite as bad though). We can fix ours by changing this function and I’m mildly confident you might be able to fix the emacs issue as well (at least I recall seeing the behaviour from that issue when I made a mistake while trying to patch refresh_multi_line).
ESS was one of the first things I looked at to contribute to when I started using Julia in Emacs. Unfortunately, I find the code difficult to understand, I think this comes from it being mostly adapted to R/S/S-plus, which was then modified slightly for other languages.
My goals aren’t really aligned with ESS - I want to support multiple editors (it looks like NeoVim/Emacs/Sublime are the main choices from my colleagues) and multiple languages (which are all implemented in Julia). So I’m trying to do as much of the work (parsing, display, debugger interface) as possible on the Julia side and keep the editor-specific code to a minimum.
I just saw that IJulia supports changing modules and they use the same REPLCompletions code that I do. If it’s possible to send new cells to Jupyter from the backend then I could use that as my display.
I am not sure I agree. ESS is based on comint, and uses a lot of tricks to capture/redirect output. These can be very fragile and get very tedious to maintain after a while. I think that a client-server approach, as suggested eg by @jamii here, is the best way to go.