Moving from a standalone script to a package for local use

Disclaimer: I’m relatively set in my ways with a decade of using R, and probably a bit spoiled by it when it comes to packages.

I currently have a working piece of Julia code as a standalone script, with a dozen functions.
I run it interactively in JuliaPro for testing, and it’s now semi-stable.
Unfortunately I don’t understand how to take the next step and make a package out of this – I’ve made one by wrapping the code as a module placed in a src/ directory, but in practice I’m still unclear how to load the package locally and reload it when I make changes (and ideally pre-compile it when things work so that startup is fast).
This area seems to be in constant motion every time I come back to this project (every few months) and I don’t know which tutorials/advice are up-to-date. Revise.jl seems promising but I’m still not sure how people would organise their workflow in practice – always keep a running session in the package’s directory, create test scripts that use Revise to reload the package when changes are made?

I wonder if there’s a Julia equivalent of https://r-pkgs.org/ to guide people like me step-by-step toward integrating packages in their workflow, with docs, tests, dependencies, etc.

See https://github.com/JuliaLang/Example.jl for an example, and https://pkg.julialang.org/docs/julia/THl1k/1.1.1/stdlib/Pkg.html for docs.

2 Likes

As for making sure that revisions are tracked: the advice that has worked for me is to develop the package (as opposed to adding it) when it is used.

I have written up what works for me here. Hope it’s useful.

4 Likes

Agree with the previous comments for the basics. Maybe for your second package you could think about using GitHub - invenia/PkgTemplates.jl: Create new Julia packages, the easy way, which can also set up CI (e.g. Travis) and documentation (Documenter.jl) stubs for you as desired, and generally gives you more options to customize. For Revise, read the docs for how to add it to startup.jl, after which you can mostly forget about it. Note that type redefinitions still require a Julia restart. Use julia --project=. in the package’s directory to use its environment. The julia-vscode plugin does project environments very nicely now, don’t know about Juno.

1 Like

Is there any workaround? (like, deleting the previous definitions maybe?)
I find restarting julia introduces enormous friction in my workflow, because it’s really sluggish (and the whole precompiled packages aspect completely eludes me).

Nope. https://github.com/timholy/Revise.jl/pull/25 is still blocked on https://github.com/JuliaLang/julia/pull/22721 afaik.

I recently did the PackageCompiler thing for a project with many dependencies and it has since saved me a lot of time. Worth learning, even though the API hasn’t stabilized yet. It works well if a lot of your dependencies don’t change: add packages you’re going to change to the blacklist and run a script that’s representative of normal usage. Everything not in the blacklist gets baked into a new system image and becomes very fast to load (but modifying those packages requires compiling a new sysimg). I use the setting "julia.additionalArgs": ["-Jsysimg.so"] (might be wrong, I’m on mobile) to use the compiled system image in vs code.

1 Like

I am happy to hear that others have a similar experience like I do.

In general I am afraid that Julia is suffering from similar problems. The reason is just that typically packages are developed by people (like you and me) who leave there projects for various reasons.
There was a great discussion about the future:
https://discourse.julialang.org/t/thoughts-on-eventual-julia-2-0-transition/

But, on the other hand, Julia has the advantage, that Julia packages are written in Julia (mostly). Not like R, where mostly every package is C (some python). If this sound exaggerating than I admit, but nearly every package you have problems with is C. So in Julia world if you have problems you can solve them more or less with Julia. Sometimes I am still concerned, because I see code in Julia, which is so advanced, that I have quite some problems to understand it, where we are somehow back in the two language problem. The two languages are now Julia + Advanced Julia.

Anyways, Julia is so much better. Do not hesitate and take the next step using modules :slight_smile:

I suppose that this distinction will disappear once you become more familiar with Julia. All powerful languages have complex layers which may take a lot of time to learn.

Compared to, say, C++, I would say that it takes significantly less effort to master almost all of Julia (the language proper), but probably still something between 2–5 years (depending on prior experience, especially with multiple dispatch). This is normal.