I’ve been using a Pluto notebook as an IDE. It’s been pretty handy being able to try things out, change definitions and try again.
Once my code is “done” though, I’d like to be able to turn it into a module/package with standard file structure.
I’ve had good luck with PkgTemplate to build the folder structure, but turning the contents of my notebook into suitable input for Documenter.jl seems way too fiddly.
Is there a tool that will read a Pluto notebook and burst it into separate Julia source and documentation files that Documenter.jl can process, and is compatible with accepted Julia package structure?
I feel having a project so structured would make it easier to add additional testing and facilitate sharing the project with others.
If there is not such a utility, I might write one. Pluto.jl provides a pretty handy API for reading through notebooks. The rest is identifying code blocks and shuffling them into appropriate files.
Please share your thoughts and suggestions about such a utility.
Yes and, I think, that that is also exactly the pain point for notebooks.
It sounds like you’re already pretty accustomed to managing complexity in software. Have you tried a text editor, a Julia REPL and Revise.jl? This latter combination scales much better.
I’m a retired computer programmer. Over half of my work experience was in lisp (Zeralisp and CommonLisp).
I have a notebook file that is over 1300 lines long. I’d rather write a utility to convert it than convert it by hand.
I do use GNU emacs when a particular editing task is too cumbersome in Pluto, either by
cutting the contents of a cell, editing in emacs, and pasting back, or by
shutting down Pluto and editing the notebook file directly.
Mostly due to my poor eyesight, I’m more comfortable using a Pluto notebook than running the REPL in an MSWindows terminal. Running the REPL in an emacs she’ll buffer is almost painful. Maybe emacs has a better mode for running a Julia REPL?
If you are starting to develop something that feels like a package, maybe you could consider an integrated development environment like Visual Studio Code. It has a lot of nice features, integrates with Revise.jl, plot panes, etc. They are also working on Pluto notebook integration so you won’t miss it.
Alternatively, just know that Pluto notebooks are normal Julia files with comments. The only thing that one would need to do to convert to a package is wrap the content of the file in a module and place in the correct place in the source directory.
It would indeed be great to be able to seamlessly transition from early prototyping in Pluto to a regular package! I asked a similar question on slack just a few days ago, but looks like nothing like this exists.
Compared to REPL or IDEs like VSCode, Pluto is reactive: tests and examples immediately reexecute when any related piece of code is changed. Also, there are no issues with modifying definitions of structs in Pluto - unlike plain julia or Revise.
You can directly import Pluto notebooks as source code in your package without converting them. The notebook itself is then the documentation, tests can stay in the notebook if they are quick, otherwise they should be moved to a separate notebook.
Parts of Pluto itself are written in this way, see e.g. Pluto.jl/Firebasey.jl at main · fonsp/Pluto.jl · GitHub .
I’ve not yet met an IDE I’ve liked, short of the MIT and Symbolics Lisp Machine environments. I feel like my software development environments have gotten progressively worse since I stopped working on Lisp machines.
I was thinking of something a bit more ambitious than Plutojl2jl.jl – collecting the markdown only cells into a single file that could be processed by Documenter.jl.
That a Pluto notebook file can be directly executed is one of the great features of Pluto. I run the notebook directly when I need line numbers to track down a bug.
Your first comment about having a 1300 line of code Pluto.jl notebook kinda scared me, but then again it is very, very nice to have the responsiveness of knowing that your code is auto-updated and you can’t have inconsistent state.
I would file an issue on the Pluto.jl - they might not work on it now, but it’s not unreasonable to respect some conventions in a notebook and then click a button and have PlutoExtras.jl or something dump that into a monorepo with tests and documentation.
a kindred spirit. I find the ide’s today noisy. For Python my “chums” make fun of me for using Idle3 or nano. I treat emacs like a controlled drug, once I start using it again it’s going to be so much fun I’ll be tweaking IT and not my code. Right now I’m using Tmux and nano ( YES! I am too lazy to use vim, sue me). My code is hand crafted with pride and OK it’s not very good but I’m getting there…
I too am liking the cut of Pluto.jl’s jib and will certainly be using it for prototyping BUT am struggling to understand how I can convert the notebook into Julia docs workflow suggestions
All that said I understand that Fons wants to get to the point where Pluto is using visual and textual programming and maybe that’s when this will come to fruition. He talks about it in this YT Fons on Pluto futures
I have been noodling over this great presentation Connor Burns Pluto NB are web api’s as a way to prototype in Pluto and deploy the results on a web page.
got to say I am so excited to be headed into 2022 and using julia for my core. Python got me coding again ( old HP high availability programmer DO NOT LAUGH!!! we did some stuff) after 10 years of not been interested. Julia has taken my coding interest to another level now I am beginning to understand dagger.jl and bought myself a 36 core 128gb ram machine. 2022 is going to be a blast! thanks to all the great people involved in julia.
Maybe emacs has a better mode for running a Julia REPL?
Check out the julia-snail emacs package. Quoting from the package’s README.md file:
Snail is a development environment and REPL interaction package for Julia in the spirit of Common Lisp’s SLIME and Clojure’s CIDER. It enables convenient and dynamic REPL-driven development.
julia-snail uses vterm for a superior terminal environment. I prefer to turn off the julia-snail completion features as described here and instead use eglot for completions, show docstrings, etc., via eglot-jl.
Edit: Just noticed that you mentioned Microsoft Windows. Snail won’t work in Windows emacs due to its dependence on vterm. I get around that limitation by installing WSL2 in windows, and then emacs, snail, and eglot-jl in WSL2/Ubuntu, running Linux emacs as a graphical application with VcXrc as the X-window manager on Windows.
Nbdev.jl 's creator here. I am kind of very late to the discussion but I feel that anyone wanting to have the following workflow Pluto notebook --> separate source code (for source file in a julia package) --> a julia package structure (project.toml etc.) --> separate documentation files (with automatic hosting on github pages) are welcome to take Nbdev.jl for a spin. it’s still in it’s infancy but I am including new features regularly.
I write most of my packages in Pluto! I don’t use a tool to convert from Pluto to something else, but the Pluto files are used directly as source code! This is a (secret) design goal of Pluto, and we have features like https://github.com/fonsp/Pluto.jl/pull/2018 to make this easier.
You can use Pluto notebooks as parts of an existing codebase, like in:
This is really useful if you have a fairly isolated piece of functionality. Pluto makes it really easy and fast to develop, document and test (with PlutoTest.jl (reactive tests with time machine)) at once.
(Large notebooks, with lots and lots of cells, are super useful for this! If you want to work on multiple parts at once, you can just open the same notebook in two windows side-by-side.)
You can also use Pluto to write all code of a package, including package tests (with CI) and documentation (CI & GitHub Pages). Examples:
The real expert in this area is @disberd, he works on packages composed of multiple Pluto notebooks with lots of cool macros and tricks to optimise his workflow.
This process definitely has some rough edges (exciting to see if Nbdev.jl can help here!) but I would totally recommend people to experience this for themselves!
