How can I avoid uselessly adding modules to an environment?

General Usage
,
1

Sorry for the confusing title, but I wasn’t sure how to say it better.

I’m currently developing some code that I’m running on an HPC. In order for the code to be easy to use and reproducible, I’ve made it into a module, and included an environment (i.e., a project.toml). When I want to compute things, I load julia with julia --project=. and load my module with using MyModule.

However I’ve found out that over time, whenever I did this and also loaded other Julia modules to experiment with new stuff, these were added to the project.toml. I now had a bunch of unused modules in my environment, which I’ve fixed using Aqua.test_stale_deps(MyModule).

How can I prevent adding useless stuff to my project.toml over time? Is there a way to avoid this? Should I just track changes to project.toml before pushing commits? Or should I instead clean up my project.toml of obsolete/unused dependencies periodically?

2

I wrote a package to help with this. It should appear in the General Registry in a day or two.

You can put the following in a script, say in ./extra/extra.jl, then do include("./extra/extra.jl")

using StackEnvs
ensure_in_stack("my_env", ["PackA", "PackB"])

When you want to use another package: 1) add it to the list, say, after "PackB". 2) Run include("./extra/extra.jl") again. (Or maybe use Revise and includet.)

Alternatively, you can look in the package to see what’s happening and do something similar yourself.

I also have found it difficult in the past, especially if I go quite a while without engaging with Julia much. One thing I kept trying to do was, 1) activate an environment with some “development” packages (that I don’t want in my default environment) 2) use APackage from that environment. 3) Re-activate the environment of the package I am working on. But Pkg does not work well that way. I think it’s not intended to work that way. (Issues arise that I won’t go into much here. To name one, if you for some reason do use APackage again, you’ll get an error because it’s not in any environment in your LOAD_PATH)

3

Just remove what you do not need any more manually.

4

This is what I’ve done in the past. It’s a chore, and requires remembering to clean up the Project.toml exactly as it was in the last commit before committing again. Even if you do it correctly, it still takes effort. And if you are still using the experimental package, you’ll have to… IDK, remove it from Project.toml, commit, push, add it back to Project.toml.

5

I mean, how many packages do you try out per week? For me it is not more than one per week. If you try out a new package you must also learn how to use it. That might need 30 min or more. Removing it from Project.toml takes 30s or less.

2 Likes
6

Just don’t install packages that you want to try out/experiment into your persistent environment(s)!

Julia has this neat feature where it suggests you to automatically install a package when doing using MyPackage. For the most convenience, add

insert!(LOAD_PATH, 2, mktempdir())

to your ~/.julia/config/startup.jl. Then, the workflow is simply

  • type using SomePackage
  • press “o”, Enter, Enter:
julia> using Accessors
 │ Package Accessors not found, but a package named Accessors is available from a registry. 
 │ Install package?
 │   (@v1.10) pkg> add Accessors 
# press "o" and Enter
 └ Select environment:
 > 1: `/var/folders/2j/9vtd991d201dbkh9dnx3wvy00000gq/T/jl_pUi3Ag` (/var/folders/2j/9vtd991d201dbkh9dnx3wvy00000gq/T/jl_pUi3Ag)
   2: `~/.julia/environments/v1.10/Project.toml` (@v#.#)
# press Enter
   Resolving package versions...
    Updating `/private/var/folders/2j/9vtd991d201dbkh9dnx3wvy00000gq/T/jl_pUi3Ag/Project.toml`
  [7d9f7c33] + Accessors v0.1.41
    Updating `/private/var/folders/2j/9vtd991d201dbkh9dnx3wvy00000gq/T/jl_pUi3Ag/Manifest.toml`
  [7d9f7c33] + Accessors v0.1.41
  [a33af91c] + CompositionsBase v0.1.2
  [187b0558] + ConstructionBase v1.5.8
  [3587e190] + InverseFunctions v0.1.17
  [1914dd2f] + MacroTools v0.5.15
  [ade2ca70] + Dates
  [de0858da] + Printf
  [4ec0a83e] + Unicode
# the package is installed and loaded

No need for any extra packages or setup, no permanent environment changes.

3 Likes
7

Doing it by hand works well for some situations. Doing this insert!(LOAD_PATH, 2, mktempdir()) etc. works well for some situations.

But if you think about it for a couple of seconds, you’ll realize that they and StackEnv are not the same in terms of function and convenience. I won’t claim, especially without thinking it through, that StackEnv is perfect for whatever workflow you have. But there are use cases, where it seems better. Like, I have a dozen packages that I have found I use when developing something. And I like the TAB completion when doing using .... Or I want to see the list of packages as a reminder. Or I appreciate the smaller amount of friction.

Or, I want to load the packages immediately, so I load this from a script.

using StackEnvs
ensure_in_stack("myenv2", [:StatsBase, :PrettyTables])
using StatsBase, PrettyTables

Not only would there be more friction if I loaded this

using StatsBase, PrettyTables

and I were prompted for which environment I wanted. In fact, it fails, saying the package can’t be found.

8

Sure, the Julia auto-install is not a universal solution for all problems :slight_smile: I don’t doubt you wrote a dedicated package because it better solves some specific issues you have.

The builtin autoinstall is just a setup-free (almost, see/support these issues) convenient way to use any packages temporarily without cluttering existing envs. Works seamlessly in my experience, and I agree autocomplete would be useful there (see the existing issue).

With a bit more setup, one can even avoid typing package names or any using/include commands for commonly-used functions. Thanks to @Lilith’s BasicAutoloads.jl.
This is my startup.jl file using this package: startup.jl with BasicAutoloads for many useful packages · GitHub.
It’s so nice to just type, eg, 123u"km/h" |> u"m/s" and it automatically installs and loads Unitful!

2 Likes