Announcing Neptune.jl (now updated to multi-line cells!)

I used Pluto for a bit off and on and the dependency analysis certainly helps, but I’d say that I found myself in a lot of situations where I wanted to quickly change something and see the effect of that change on just a couple cells without propagating the change the whole way through the notebook. One can do things like comment out cells, or introduce execution barriers, but I guess I’m just already quite accustomed to non-reactive notebooks and I never thought the state fullness was such a big problem for me (but I also rarely have notebooks with more than like 20 active cells because I factor things out into private packages).

But reactivity is also quite nice and has its own advantages of course. There’s lots of tradeoffs to be made here

9 Likes

Just because something is correct, doesn’t mean that it is desired.

1 Like

One downside of Neptune is that Pluto’s save files do not store the outputs of the cells. This can become a problem if the outputs depend on the order you execute the cells in. After saving and reopening a notebook, you might be unable to reproduce the outputs you had before.

So, I think you should think of a way to store outputs in the save file somehow. If you figure out a good way (without losing the “notebooks as code” idea), I think that it might be nicer to add ways to conveniently use such a format to Jupyter (rather than making a new tool).

1 Like

For saving output, I use the pdf printing feature of the browser, but it would be nice to be able to convert to/from ipynb. That would be a big undertaking, though. At this stage, I see Neptune as a handy, simple, lightweight tool (at least my students have found it so). But anyone who wants to submit a pull request is welcome!

1 Like

Fair enough. I take it, the reason you made Neptune is that your students found the reactivity of Pluto confusing?

No, I originally did it for myself, because I saw Pluto as being a really neat way to program in Julia, but (like many others) my programming process is sequential, and for Data Science (as others have explained in this forum and elsewhere) reactivity seems problematic (to say the least). Because of my needs, reactivity is an intriguing novelty but no more. For others, reactivity seems to be a mission, which is fine - I am all in favour of diversity of approaches and thought (unlike the lead developer of Pluto), I think people like myself should have a choice. I would have preferred that what is called Neptune would be a mere option in Pluto, but it was made clear to me that this is never going to happen. So Neptune gives us a choice. Period. The question of ‘which is better’ is entirely personal and situational.

13 Likes

Hey compleat,

Can you be more kind and less personal?
I feel a lot of aggression here and even though I want to share some thoughts, I feel unsafe to do so.

Also, the fact that you took a piece out of context from a private (“but not confidential”) conversation and posted on a public forum is something I don’t like very much.

Can you clean this thread from personal attacks so I can participate too?

13 Likes

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

post removed in good faith

1 Like

This thread is quickly diverting :slight_smile: let’s try and keep it on topic

Edit: and please avoid personal attacks or the like. Saying that you don’t agree with someone is fair, saying that someone is menace based on one interaction is not (IMO), regardless, let’s keep this friendly and civil.

12 Likes

Thanks, I agree

1 Like

Quote removed

3 Likes

A post was split to a new topic: On moderation and edits

To be clear: I deleted my earlier quote not because I was censored or because I wanted to retract anything I said, but because it (the quote) seemed to be distracting the thread from being constructive.

I created Neptune (because (like so many others julia-ites, I think) I was blown away by the elegance of Pluto when I first saw it, but subsequently astounded that the developers were resolute in their refusal to take it the one step further to make it truly useful, which would have been to allow (as an option) sequential evaluation. I then started working on Neptune and funded it myself (without support or reimbursement of any kind) in order to make it available to my students and others (particularly data scientists). I did not do it out of any spiteful or aggressive motives, but rather just out of desire to help out the Julia community. To those who are critical of, or hostile towards Neptune, I have a great suggestion: just ignore it and let others use it (or not).

Anyway, I hope some of the community find the package useful. Thank you for your feedback. Further feedback and suggestions (hopefully constructive) can be be given on the repo. Thanks

19 Likes

The discussions here remind me of something that’s occurred to me a few times. I’ve often used a notebook (both Jupyter and Pluto) to do some exploration, and then later regretted it somewhat. In the case of Jupyter because of the out-of-order pitfalls that are well known, and in the case of Pluto because while the reactivity is a really awesome feature, I’ve found it can be a bit difficult to factor out code that belongs in an extra-notebook form (file to be include’d, or even a package). Usually it seems easier to just start with another blank notebook and manually put back only what needs to be there to create the ideal reactive document. That’s not a knock on Pluto at all, BTW, just a confession that I’m probably abusing it!

By contrast the vanilla REPL doesn’t have the reactivity ‘potential for abuse’ of Pluto or the out-of-order problems of Jupyter, but of course lacks many of the nice features of both Jupyter and Pluto (mini-IDE in each cell with syntax highlighting, etc; ability to have rich text annotations, even including LaTeX; inline interactive plots; …).

Anyway. My point actually isn’t that either Jupyter or Pluto should change much, or at all, but that I wonder if there is kind of another complementary approach that could be taken.

What if there was something that behaved like the REPL in that you could not execute things ‘out of order’ but only recall previous entries as you can with the REPL with the up arrow (and Ctrl+R etc). Basically simply Jupyter but with only the ability to append cells and allow the same kind of workflow as I think a lot of us do in the REPL:

julia> f(x) = x + 2  # first swing at how f should work
f (generic function with 1 method)

julia> f(2)
4 # wait, that's not right...

julia> f(x) = x - 2  # oh yeah it should have been this instead
f (generic function with 1 method)

julia> f(2)
0  # hmm, that's still not right ...

.
. # rinse, repeat..
.

julia> f(x) = x^3
f (generic function with 1 method)

julia> f(2)
8 # that's it, time to copy and paste f into a .jl file!

At this point, especially in the creation of ‘scripts’ (more so the users, especially novice, of Julia as opposed to, typically more experienced, package developers), a common pattern would be to copy-paste one’s terminal history to a file and then manually scroll through and remove lines that were dead-ends, likely making mistakes in doing so, resulting in going back to the REPL and manually replaying the process etc,.

So imagine each julia> prompt is more like a Jupyter or Pluto cell with all the mentioned advantages, but where you can press the up arrow to recall, change, and re-execute.

Now where I wonder if the great technology that Pluto has developed could come in: what if this tool could automagically flag prior cells that become invalid because of later entries? Such that you could just press a button and have such now-invalid cells are removed, and get out a minimal history resulting in the final workspace, with whatever methods have resulted and global scope results that have resulted. Then you have reproducibility (another Pluto advantage) but always being able, and indeed required, to be read top-to-bottom, Which addresses, I think, some of the concerns of using both Pluto and Jupyter, especially in some teaching contexts (including self-teaching).

In terms of implementation (and to be clear I personally am nowhere near skilled enough to do so so this is so far maybe at best a thought experiment) clearly this would be best implemented by taking parts of both Jupyter and Pluto and factoring these out to separate packages depended-on by both of these tools, which of course would take some effort and involve a certain amount of cooperation on the part of maintainers of both projects, but at least wouldn’t have to involve an all-out fork of either project, the downside of which I think are what’s behind some of what is being discussed here and elsewhere in regards to the OP’s project.

So, just an idea, which maybe has been substantially proposed and maybe even in part done before (some Juno/VS Code functionality intersects) but I thought I’d put it out there.

Finally, I think every post about Pluto, Jupyter, and such tools that make Julia more accessible and useful should conclude with sincere thanks to the authors of these! :pray: :open_hands: :slightly_smiling_face:

4 Likes

I used Jupyter quite a lot for data analysis and (recently) also Pluto. Having some rather long-running cells, the reactivity in Pluto has a price, as mentioned above. On the other hand, reactivity removes issues with out-of-order execution and makes a great interactive GUI for exploration.
It would be great to keep reactivity, but to find a way to mitigate the issues with long-running cells. In my opininon, adding the following functionalities to Pluto would do it:

  1. Display the cell dependencies and execution order in Pluto GUI: https://github.com/fonsp/Pluto.jl/issues/13
  2. The temporary execution barrier in https://github.com/fonsp/Pluto.jl/discussions/298

This way, you could “switch off” your long-running cells and only execute them on demand while keeping reactivity. With the dependencies visible in the notebook issues with “accidentially” updating cells should be easy to spot and fix.

Edit: with 1. you could order your cells in the notebook according to execution order. Together with 2. you get essentially a Jupyter-style notebook.

3 Likes

You forget, that one big advantage of Neptune over Jupyter is that the saved files can be loaded as Julia scripts. I would not be in favour of messing that up.

(FWIW)

4 Likes

Yes. If you executed the saved .jl as a script, it would eexcute in the order presented. Anyway, if you don’t understand the relevance, then don’t worry about it - just that I have my reasons for appreciating/noting your suggestion only, but not pursuing it.

I asked a technical question, after trying out this package. I think this is an interesting space to explore, and I presumed such discussion was the whole point of announcing it here. But if this is the desired tone, then I’m out.

12 Likes

It’s interesting to see that people generally like the properties of a reactive notebook but want some way around it to avoid costly updates to propagate while iterating on their ideas. Recently I saw a very nice talk about a very original IDE/computation environment called Dion. One of the features they showcase is this idea of “code slices” – basically you click on a piece of code and the editor will show you only other parts of code that interact with it, sort of a connected component (you can see a demo here Dion Systems - The How And Why Of Reinventing The Wheel on Vimeo at 41:23).

Now here are my 2 cents, tossed from my comfy armchair: Say you want to update some cell A in Pluto notebook, so you request a slice of dependencies of A and Pluto would show them to you, maybe in a split screen, as a separate notebook. In this separate notebook, you can see that A triggers cells B and C, but you are only interested of As effect on B, so you delete the cell C in your new sub-notebook. Once you are happy with A and B, you close the view and the cells A and B will get updated in the original notebook.

Seems like Pluto already tracks a lot of the metadata needed for this, but it’s still a lot of work so feel free to ignore the idea. I’d definitely recommend watching the video above, though, those guys are pretty creative.

7 Likes