Announcing Neptune.jl (now updated to multi-line cells!)

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

post removed in good faith

1 Like

This thread is quickly diverting :slight_smile: let’s try and keep it on topic

Edit: and please avoid personal attacks or the like. Saying that you don’t agree with someone is fair, saying that someone is menace based on one interaction is not (IMO), regardless, let’s keep this friendly and civil.

10 Likes

Thanks, I agree

1 Like

Quote removed

3 Likes

A post was split to a new topic: On moderation and edits

To be clear: I deleted my earlier quote not because I was censored or because I wanted to retract anything I said, but because it (the quote) seemed to be distracting the thread from being constructive.

I created Neptune (because (like so many others julia-ites, I think) I was blown away by the elegance of Pluto when I first saw it, but subsequently astounded that the developers were resolute in their refusal to take it the one step further to make it truly useful, which would have been to allow (as an option) sequential evaluation. I then started working on Neptune and funded it myself (without support or reimbursement of any kind) in order to make it available to my students and others (particularly data scientists). I did not do it out of any spiteful or aggressive motives, but rather just out of desire to help out the Julia community. To those who are critical of, or hostile towards Neptune, I have a great suggestion: just ignore it and let others use it (or not).

Anyway, I hope some of the community find the package useful. Thank you for your feedback. Further feedback and suggestions (hopefully constructive) can be be given on the repo. Thanks

16 Likes

The discussions here remind me of something that’s occurred to me a few times. I’ve often used a notebook (both Jupyter and Pluto) to do some exploration, and then later regretted it somewhat. In the case of Jupyter because of the out-of-order pitfalls that are well known, and in the case of Pluto because while the reactivity is a really awesome feature, I’ve found it can be a bit difficult to factor out code that belongs in an extra-notebook form (file to be include'd, or even a package). Usually it seems easier to just start with another blank notebook and manually put back only what needs to be there to create the ideal reactive document. That’s not a knock on Pluto at all, BTW, just a confession that I’m probably abusing it!

By contrast the vanilla REPL doesn’t have the reactivity ‘potential for abuse’ of Pluto or the out-of-order problems of Jupyter, but of course lacks many of the nice features of both Jupyter and Pluto (mini-IDE in each cell with syntax highlighting, etc; ability to have rich text annotations, even including LaTeX; inline interactive plots; …).

Anyway. My point actually isn’t that either Jupyter or Pluto should change much, or at all, but that I wonder if there is kind of another complementary approach that could be taken.

What if there was something that behaved like the REPL in that you could not execute things ‘out of order’ but only recall previous entries as you can with the REPL with the up arrow (and Ctrl+R etc). Basically simply Jupyter but with only the ability to append cells and allow the same kind of workflow as I think a lot of us do in the REPL:

julia> f(x) = x + 2  # first swing at how f should work
f (generic function with 1 method)

julia> f(2)
4 # wait, that's not right...

julia> f(x) = x - 2  # oh yeah it should have been this instead
f (generic function with 1 method)

julia> f(2)
0  # hmm, that's still not right ...

.
. # rinse, repeat..
.

julia> f(x) = x^3
f (generic function with 1 method)

julia> f(2)
8 # that's it, time to copy and paste f into a .jl file!

At this point, especially in the creation of ‘scripts’ (more so the users, especially novice, of Julia as opposed to, typically more experienced, package developers), a common pattern would be to copy-paste one’s terminal history to a file and then manually scroll through and remove lines that were dead-ends, likely making mistakes in doing so, resulting in going back to the REPL and manually replaying the process etc,.

So imagine each julia> prompt is more like a Jupyter or Pluto cell with all the mentioned advantages, but where you can press the up arrow to recall, change, and re-execute.

Now where I wonder if the great technology that Pluto has developed could come in: what if this tool could automagically flag prior cells that become invalid because of later entries? Such that you could just press a button and have such now-invalid cells are removed, and get out a minimal history resulting in the final workspace, with whatever methods have resulted and global scope results that have resulted. Then you have reproducibility (another Pluto advantage) but always being able, and indeed required, to be read top-to-bottom, Which addresses, I think, some of the concerns of using both Pluto and Jupyter, especially in some teaching contexts (including self-teaching).

In terms of implementation (and to be clear I personally am nowhere near skilled enough to do so so this is so far maybe at best a thought experiment) clearly this would be best implemented by taking parts of both Jupyter and Pluto and factoring these out to separate packages depended-on by both of these tools, which of course would take some effort and involve a certain amount of cooperation on the part of maintainers of both projects, but at least wouldn’t have to involve an all-out fork of either project, the downside of which I think are what’s behind some of what is being discussed here and elsewhere in regards to the OP’s project.

So, just an idea, which maybe has been substantially proposed and maybe even in part done before (some Juno/VS Code functionality intersects) but I thought I’d put it out there.

Finally, I think every post about Pluto, Jupyter, and such tools that make Julia more accessible and useful should conclude with sincere thanks to the authors of these! :pray: :open_hands: :slightly_smiling_face:

4 Likes

I used Jupyter quite a lot for data analysis and (recently) also Pluto. Having some rather long-running cells, the reactivity in Pluto has a price, as mentioned above. On the other hand, reactivity removes issues with out-of-order execution and makes a great interactive GUI for exploration.
It would be great to keep reactivity, but to find a way to mitigate the issues with long-running cells. In my opininon, adding the following functionalities to Pluto would do it:

  1. Display the cell dependencies and execution order in Pluto GUI: Visualize cell reactive dependencies · Issue #13 · fonsp/Pluto.jl · GitHub
  2. The temporary execution barrier in Feature idea: a temporary execution barrier · Discussion #298 · fonsp/Pluto.jl · GitHub

This way, you could “switch off” your long-running cells and only execute them on demand while keeping reactivity. With the dependencies visible in the notebook issues with “accidentially” updating cells should be easy to spot and fix.

Edit: with 1. you could order your cells in the notebook according to execution order. Together with 2. you get essentially a Jupyter-style notebook.

3 Likes

You forget, that one big advantage of Neptune over Jupyter is that the saved files can be loaded as Julia scripts. I would not be in favour of messing that up.

(FWIW)

2 Likes

Yes. If you executed the saved .jl as a script, it would eexcute in the order presented. Anyway, if you don’t understand the relevance, then don’t worry about it - just that I have my reasons for appreciating/noting your suggestion only, but not pursuing it.

I asked a technical question, after trying out this package. I think this is an interesting space to explore, and I presumed such discussion was the whole point of announcing it here. But if this is the desired tone, then I’m out.

10 Likes

It’s interesting to see that people generally like the properties of a reactive notebook but want some way around it to avoid costly updates to propagate while iterating on their ideas. Recently I saw a very nice talk about a very original IDE/computation environment called Dion. One of the features they showcase is this idea of “code slices” – basically you click on a piece of code and the editor will show you only other parts of code that interact with it, sort of a connected component (you can see a demo here Dion Systems - The How And Why Of Reinventing The Wheel on Vimeo at 41:23).

Now here are my 2 cents, tossed from my comfy armchair: Say you want to update some cell A in Pluto notebook, so you request a slice of dependencies of A and Pluto would show them to you, maybe in a split screen, as a separate notebook. In this separate notebook, you can see that A triggers cells B and C, but you are only interested of As effect on B, so you delete the cell C in your new sub-notebook. Once you are happy with A and B, you close the view and the cells A and B will get updated in the original notebook.

Seems like Pluto already tracks a lot of the metadata needed for this, but it’s still a lot of work so feel free to ignore the idea. I’d definitely recommend watching the video above, though, those guys are pretty creative.

6 Likes

Yes @compleat, you should do you! Congratulations on your package, which seems like a very good idea. But this discussion might give some insight on Pluto.jl. Everything about that package strikes me as one person’s grand vision. It’s quite a remarkable and novel idea. It’s also an experiment, and the developer is taking their idea and seeing how far it can go. Sometimes purity of vision is important for an experiment to be falsifiable. If you were

and are

Just as you in response to a suggestion, maybe they

Even if you believe

we should salute them, just as we salute you. You have provided valuable insight on the workflow and needs of the data scientist, and demonstrated that ideas from Pluto.jl might well be taken elsewhere. Open source software allows us to explore many different approaches, some developed by one collaborative group, others forked and taken in a new direction. Thank you (and Pluto.jl) for sharing with everyone.

40 Likes

What about a dual windows/tabs IDE where on one side you have

  • a DAG of cells (based on their time lines) where one can manipulate visibility, (re-)evaluation, reactivity on/off, reorder (with a limit based on cells dependencies), etc.

And the other side

  • A cell editing windows with previous version/evaluation available. Possibly several cells could be automatically available (based on selection and dependencies to other opened cells)

This is a very rough idea it could combine pretty well with exporting to various format, and adapt to the needs of the user.

EDIT: I didn’t want to hijack Neptune.jl ANN, but it could be a future path for such an IDE :stuck_out_tongue:

Well said!

I am not against Neptune being extended - as long as the extensions would not distract from the simplicity of the basic functionality. For me, the simplicity of the ‘default’ experience (and that includes not having too many distracting or confusing choices) is paramount - I would like it to be useable by elementary undergraduates and absent-minded types (me?) alike. If there are (almost unnoticeable) options which lead to much greater power to expert (and intellectually energetic) users, so much the better! [Thanks for the comments/ideas]

I’ll be just waiting for a moment when someone announces Saturn.jl that will have optional reactivity. There is no right way — reactive or not, I think it should just be togglable and up to a user to decide on the fly.
:sweat_smile:

8 Likes

I have a neptune notebook now, and it is all pretty - the uninteresting code is hidden, text is markdown etc. When I reopen the notebook, and want to run all the code I currently only see the option of manually un-hiding code in each cell, run it, and re-hide the code. That kind of defeats the purpose - is there a way of rerunning the entire thing, or better yet, a section, in a more automatic fashion?

Try
Neptune.run(run_notebook_on_load=true)

This is supposed to work, but I haven’t used it.