Developing Julia on a moderate laptop is very tedious

[schlichtanders]

Hi there,

I like to gather how other people develop julia packages. I so frequently run into the point where I changed a struct field name or const and need to restart julia, with compilation times which make me a bit sad.

It would be awesome to either have Revise for everything or a reasonable interpreter mode for everything (like turning off compilation somehow, does not need to be entirely, but so that at least development of your own package is quick).

How are others developing somewhat larger packages with compilation times of 2+ minutes on moderate laptops?

[Oscar_Smith]

one thing to consider is launching Julia with -O1. it will significantly reduce compile time

[co1emi11er]

Have you tried PlutoDevMacros.jl?

It changed the way I develop packages. You can watch this video to get an idea. https://youtu.be/eHRURW6Wfpc?si=92x7Ch-josFGnS7D

[littlelib]

Not sure if it’s any good, but when I’m sure I’ll have to meddle with struct name or fields quite often, I put it in a separate module and replace the module when necessary.

[ufechner7]

I try to avoid const globals and use typed globals instead. Using Linux on a fast laptop (Ryzen 7840U) also helps. Furthermore I use a custom system image that contains all the packages my package depends on such that restarting Julia and and using the package I am working on does not take longer than 5s.

Finally I sometimes comment the precompile statements of my package which can also save some time during the development.

[Sevi]

Specifically for changing structs often, there is also ProtoStructs.jl

I found it most useful so far when creating completely new types, but it should also work to just put @proto in front of existing structs, recompile once, develop, and then remove @proto at the end.

[MilesCranmer]

This looks super cool as a solution to the struct issue. I can’t seem to figure it out though… Is it supposed to work in normal Julia runtimes, or just within Pluto?

When I tried it for developing SymbolicRegression, using the example at 14:31 in the video, I get this error:

# Started with `julia --project=.` in the root of SR

julia> using PlutoDevMacros: @fromparent

julia> @fromparent import *
ERROR: LoadError: AssertionError: The extracted path does not seem to be a valid path.
-`extracted_path`: /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/SymbolicRegression.jl/REPL[2]

julia> @frompackage "." using SymbolicRegression
# No error, but no symbols are actually imported

Tried about 10 different variations but doesn’t seem to work. I’m probably doing something dumb but I could not seem to figure it out from the talk or docs.

[MilesCranmer]

Is there an environment variable for setting default optimization levels or would I need to set up an alias for this? I just realised the environment variable page doesn’t say anything about this.

[co1emi11er]

I just use it within Pluto. I think it uses Pluto’s ability to redefine structs. You can take a look at TapeMeasure.jl which is one of my most recent projects. I just have two Pluto notebooks in the src folder dim_pluto.jl and dim_pluto_makie.jl and I think all they have at the beginning is just:

using PlutoDevMacros
@fromparent import *

Then all my structs and functions are available inside pluto to mess around with even if I don’t export them which I find pretty great in the early stages.

[MilesCranmer]

Thanks. Pluto is great but there are just way too many useful extensions in VSCode/Cursor for me ever want to leave that ecosystem. But hopefully Pluto can one day actively support Pluto.jl (alpha preview) - Visual Studio Marketplace though!

[co1emi11er]

I code in vscode. I just have a Pluto notebook open on the side. None of my actual code lives in there. I just save my file i am editing in vscode and I see the updates on the Pluto notebook

[MilesCranmer]

I see. That’s an interesting workflow, I will try it out - thanks!

[roflmaostc]

I heard about this trick too but with my vim + Linux + Firefox I never got it to work.

Updates from Browser propagate into vim. But not from vim into the browser.
Also, VSCode nor vim would insert new cells, right?

[Mason]

@roflmaostc

Updates from Browser propagate into vim. But not from vim into the browser.

when you launch Pluto you need to do

Pluto.run(;auto_reload_from_file=true)

@roflmaostc

Also, VSCode nor vim would insert new cells, right?

Don’t know about them, but I know emacs can with pluto-mode: GitHub - torfjelde/plutojl-mode.el

Can’t be that hard to do in other editors too

[Palli]

I think Julia should have a --dev[eloper] mode with all the good options below, e.g. disable const and implying --debug-mode (Julia does have some debug mode but only for logging seemingly, I want more), and I suppose also --project=. (I wish I knew of this sooner).

@schlichtanders

It would be awesome to either have Revise for everything or a reasonable interpreter mode for everything

I think you have it in effectively with Revise.jl plus ProtoStructs.jl and with running with --inline=no which is I think implied by:

@Oscar_Smith

one thing to consider is launching Julia with -O1. it will significantly reduce compile time

I’m not sure what else changes with -O1 or -O3 really like to know, but I believe inlining is the major problem, i.e. recompilations, which inlining induces. Without inlining, compilation speed even with (otherwise) -O3 should be too much of a problem I think. There’s even --compile=min for a full interpreter, but likely overkill.

ProtoStructs.jl referes to likely outdated alternative Redef, actually RedefStructs.jl. I don’t know the differences but I guess both must add extra redirection (whick slows down a bit at runtime, but makes development time faster/easier).

@ufechner7

I try to avoid const globals and use typed globals instead.

How would you like a mode (or just a package, to begin with)that ignores it (or only for globals)? Synce this is syntax, seemingly can’t be changed in a package, but I think that would even be false… The new parser is in JuliaSyntax.jl, and it was once stated from the REPL dynamically (while it was in development), so since that’s possible basically anything can be redefined at runtime, i.e. to me it seems possible a package could use a fork of it that adds @proto to struct definitions and drops const.

@littlelib

Not sure if it’s any good, but when I’m sure I’ll have to meddle with struct name or fields quite often, I put it in a separate module and replace the module when necessary.

@co1emi11er

Have you tried PlutoDevMacros.jl?

No, just discovered it now:

@MilesCranmer

Is it supposed to work in normal Julia runtimes, or just within Pluto?

Seemingly, I see no dependency on Pluto or related. I see a dep on JuliaInterpreter.jl though. If/since not Pluto-specific, maybe the package should be renamed to DevMacros or something, just Dev? It seems to overlap with other packages allowing redefining structs (without something like @proto?). A Dev.jl package could depend on it and all the good ideas that help here, and end up as a stdlib. A package alone can’t change (global) optimization setting though, so it alone can’t do what I propose with --dev. [You can though change optimization locally for modules only I’ve seen so far, though I would want more finegrained, and allow applying to calling site, like you can now for some macros like @inline.]

@ufechner7

Finally I sometimes comment [out] the precompile statements

Interesting, yes, they help users with speed (after install), but not developers, maybe for them precompile can and should be redefined to no-op? Could be done in a package and the -dev mode I proposed.

@MilesCranmer

Is there an environment variable for setting default optimization levels or would I need to set up an alias for this?

I don’t think so, not seen it, might have missed it, but also I think I wouldn’t want this. Seems easy to misremember is applied (just like --startup-file=yes is easily forgotten, shouldn’t be on be default, only for the REPL), and alias juliad (and juliac) seem better, more explicit.

[roflmaostc]

wow, that worked

GitHub - hasundue/vim-pluto: Vim plugin for editing Pluto notebooks

[co1emi11er]

I just meant I do something like this:

[schlichtanders]

Thanks for the suggestion. A --dev flag sounds like a really nice suggestion, especially since all pieces seem to be already there.

What would be the procedure to make this an official suggestion?

[Benny]

I don’t think there will be an entirely satisfying approach, at least I haven’t run into one.

Moving from compilation to interpretation sacrifices performance, and in some cases that does make development annoyingly slow e.g. Debugger.jl’s toggled compiled mode. I also tend to check type inference often, and not needing recompilation upon type changes or messing with consts, including the implicit ones for type and function definitions, will make that impossible. While redefining methods also need wider code changes to the calls, type redefinitions have the extra headache of lingering obsolete instances, subtyping, and annotations. This issue occurs in Python, an interpreted language with effectively non-const definitions, and it’s only fixed by handpicked or complete reevaluation there too.

ProtoStructs.jl is actually pretty cool because it doesn’t mess with const-ness at all, so type inference sometimes works normally. Fair warning though, ProtoStructs doesn’t work on v1.11.1 for me right now because of some missing Base internal. It basically transforms a type definition into a iterated union of types wrapping a NamedTuple for your fields (a structural type), and it’s the type constructors that get redefined for a particular “redefinition’s” NamedTuple. Type annotations in methods accept any “redefinition” as a result, though the now abstract type doesn’t do as well in other struct’s field type annotations. “Redefinitions” in ProtoStructs still need recompilation of methods, and that also goes for Pluto, which works by re-evaluating all related cells in new hidden modules. Recompilation is only not an issue if there wasn’t much to do in the first place, but at least we don’t have to restart the Julia runtime. Great for smaller stuff in early stages, but it’s better to be more certain of a type’s fields before the methods and compilation times accumulate.

[disberd]

Hi @co1emi11er,

Nice to see that PlutoDevMacros is actually useful to someone else :).

This is also the workflow I mostly use now for developing packages, mostly coding from Cursor and then also changing pluto notebook code from Cursor and exploit the auto_reload_from_file functionality to propagate changes to the browser.

I had also developed a very basic VSCode extension to simplify deleting/adding cells from within the editor itself, which can be found here:

Pluto Operations - Visual Studio Marketplace

@MilesCranmer PlutoDevMacros does indeed only work with Pluto at the moment as it heavily relies on its ability to redfine structs seamlessly.

That being said, I was just thinking the other day about making a package to achive something very similar also in the REPL by levearging the rather new functionality to switch active module within the REPL (mostly to do the dev work inside a temp module that is not Main to simplify resetting the module in case of a struct change).
I hope to be able to get a working prototype of this in the next 2-3 weeks!

I know I still have to update the documentation of the package after JuliaCon changes but feel free to ping me or write an issue in case you need some more guidance on how to use it in the meanwhile