Tip: macro to install/use package in temporary environment

General Usage
1

I have been using this pattern, which loads a package from a temporary environment (to not add it to the current one):

julia> ] activate --temp

(jl_3FBlLK) pkg> add Plots

julia> using Plots

julia> ] activate .

julia> plot(rand(10))

such that I can use Plots without adding it to any permanent environment (or other package).

Now I’ve added this simple macro to my startup.jl file:

#
# use with: @using_temp Plots
#
macro using_temp(pkgname)
    using_expr = Expr(:using)
    using_expr.args = [Expr(:., pkgname)]
    esc(quote
        const cpp = $Pkg.project().path
        try
            import Pkg
            $Pkg.activate(temp=true, io=devnull)
            $Pkg.add($("$pkgname"))
            $using_expr
        finally
            $Pkg.activate(cpp, io=devnull)
        end
    end)
end

With which I can do:

julia> @using_temp Plots

julia> plot(rand(10))

with the same effect.

I found that useful. Maybe the macro can be improved (I’m very unfamiliar with writing macros).

Any comment is welcome.

8 Likes
2

Julia automatically installs a package when you try to load it:

julia> using Plots
 │ Package Plots not found, but a package named Plots is available from a registry. 
 │ Install package?
 │   (@v1.9) pkg> add Plots 
 └ Select environment:
 > 1: `/tmp/jl_Y5S4mT` (/tmp/jl_Y5S4mT)
   2: `~/.julia/environments/v1.9/Project.toml` (@v#.#)
   Resolving package versions...
<...>
julia> # Plots is loaded

so there’s probably not much to be gained by such a macro. By default, it only asks to install the package into the global or current env, in order to get the temp env automatically you need to add this to startup.jl:

insert!(LOAD_PATH, 2, mktempdir())

Also, relatively recently there appeared an idea of “auto-loading” a package when something from it is used in the REPL. Combining these two features (auto-load and auto-install) makes working in the REPL even more convenient:

# fresh Julia start
julia> SVector(1, 2)
[ Info: Loading StaticArrays for SVector ...
 │ Package StaticArrays not found, but a package named StaticArrays is available from a registry. 
 │ Install package?
 │   (@v1.9) pkg> add StaticArrays 
 └ Select environment:
 > 1: `/tmp/jl_Xo2l9A` (/tmp/jl_Xo2l9A)
   2: `~/.julia/environments/v1.9/Project.toml` (@v#.#)
   Resolving package versions...
<...>
2-element SVector{2, Int64} with indices SOneTo(2):
 1
 2

See my take on the startup.jl with auto-load and auto-install at startup.jl · GitHub. There are others as well.

4 Likes
3

I don’t think these things solve what I use the macro for. I do not want the package to be installed in any persistent environment, but I want to use it temporarily in the environment I’m working at.

Am I missing something?

So, if I’m developing something in a package environment:

MyPkg>

and want to plot something, I do:

julia> my_package_generated_data = MyPkg.f()

julia> @using_temp Plots

julia> plot(my_package_generated_data)

and I do not mess up with any environment with that.

That works nicely now that native code is cashed, because the @using_temp package does not get installed from zero every time.

ps:

I also have this:

Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true

in my startup.jl file, and that probably makes things even faster for that. I guess that could be put in the macro directly.

3 Likes
4

Ps: My Julia doesn’t ask in which environment I want to install a package. Where that comes from?

5

Exactly, that’s the scenario I demonstrated. The package is installed into a temp env that is created at the same time, and forgotten after exiting from REPL.

No matter if starting a regular REPL (in the global env) or in the project env — no actual persistent env is modified!

Not sure if it’s documented anywhere other than in a changelog, but since 1.7 Julia offers to install packages automatically on using: Julia v1.7 Release Notes · The Julia Language.
So that you see a temp environment in the list of envs to install, add insert!(LOAD_PATH, 2, mktempdir()) to your startup.jl.

6

Yes, but for me it installs in the active environment without giving me a choice. That’s what I’ve never seen (and would make the macro clearly less useful). I’m on 1.10 here, so either you have nightly there or some option or package installed that gives you that choice.

7
  • Run julia> using Plots
  • It shows this prompt:
julia> using Plots
 │ Package Plots not found, but a package named Plots is available from a registry. 
 │ Install package?
 │   (@v1.9) pkg> add Plots 
 └ (y/n/o) [y]:
  • Type o<Enter>
  • Environment selection appears:
julia> using Plots
 │ Package Plots not found, but a package named Plots is available from a registry. 
 │ Install package?
 │   (@v1.9) pkg> add Plots 
 └ Select environment:
 > 1: `/tmp/jl_ZBaVNE` (/tmp/jl_ZBaVNE)
   2: `~/.julia/environments/v1.9/Project.toml` (@v#.#)
  • Press Enter
4 Likes
8

Hehe, cool. I’ve never noticed that “o” before…

Yet, still, it returns to the environment where the package was installed, not to the previous one. So it will have to be completed by “activate [previous environment]”, which makes the macro slightly better.

(I would be totally in favor of using Package to - by default - install the package in a temporary environment).

9

I think the original example by @aplavin wouldn’t need to activate the temp ENV because it is added in the LOAD_PATH from the command

insert!(LOAD_PATH, 2, mktempdir())

In the original reply

10

Yes, but after installing the package you end up with that temporary environment enabled.

11

Ah I missed that from your answer above, sorry

12

Why is that important? You previous env remains in the load path, packages can still be used from it.

13

I think this final environment being the temporary one only happens if the temporary environment is at the top of the load path. If you’re activating a project, it’ll supersede the temporary environment. This is something that I use very frequently

14

Well, because if I’m developing the package in that environment, I’m expecting that the other commands, like add, rm, act on it. My goal is just to have a package loaded without affecting other workflows.

I can’t test that now, but maybe the environment at the top of load path ends up being loaded, which is not ideal in any case, I just want to have the previous environment loaded, as if nothing had happened.

15

Perhaps I wasn’t precise, but this is exactly what happens.

(@v1.9) pkg> activate a
  Activating project at `~/a`

julia> insert!(LOAD_PATH, 1, "b")
5-element Vector{String}:
 "b"
 "@"
 "/tmp/jl_55Zdxa"
 "@v#.#"
 "@stdlib"

julia> using FillArrays
 │ Package FillArrays not found, but a package named FillArrays is available from a registry. 
 │ Install package?
 │   (a) pkg> add FillArrays 
 └ Select environment:
   1: `~/b/Project.toml` (b)
   2: `~/a/Project.toml` (@)
 > 3: `/tmp/jl_55Zdxa` (/tmp/jl_55Zdxa)
   4: `~/.julia/environments/v1.9/Project.toml` (@v#.#)
   Resolving package versions...
    Updating `/tmp/jl_55Zdxa/Project.toml`
  [1a297f60] + FillArrays v1.6.1
    Updating `/tmp/jl_55Zdxa/Manifest.toml`
  [1a297f60] + FillArrays v1.6.1
  [56f22d72] + Artifacts
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [e66e0078] + CompilerSupportLibraries_jll v1.0.5+0
  [4536629a] + OpenBLAS_jll v0.3.21+4
  [8e850b90] + libblastrampoline_jll v5.8.0+0

(a) pkg>

As you see, I installed a package in a temp environment lower in the load path, but the currently activated environment was unchanged. This is irrespective of what’s above it in the load path. Frequently, the topmost element is the activated environment, but this doesn’t need to be the case.

1 Like
16

Ok, so the suggestion of @aplavin is an alternative, as he described:

  1. add insert!(LOAD_PATH, 2, mktempdir()) to .julia/config/startup.jl. The 2 there is important.

  2. When using Plots, type “o” and this will prompt which environment you want it to be installed. The good thing is that by default the temporary environment is chosen.

Here the thing is tricky (and I could not find any documentation describing this behavior):

2.1. If a custom environment was activated before using, this environment will be activated after the package is loaded (which arguably is what we want).

2.2. If the global main environment 1.10 Pkg> is activated, the environment active after the package is loaded is the temporary environment (which may or not be what one wants).

Anyway, this is an alternative to the macro above. I’m not sure how reliable this behavior is across versions, as I couldn’t find the documentation associated with it.

17

It works in Julia 1.9.3 which is the recommended version for most people. I wouldn’t be surprised if it worked for 1.7 and above.

18

Often it doesn’t matter which env is activated after installing a temp package. But yes, when it matters — would be more convenient if it was the previous actually activated env, not the temp one.

This is hard if possible to reliably achieve within the current y/n/o behavior and LOAD_PATH manipulations. Recently, @stefan suggested it may be possible to specifically support temp env in this dialog:

We could add a t option so that it’s y/n/o/t (why not, indeed). The t option would add a temp directory to the load path if there isn’t one already (presumably determined by whether it starts with /tmp or wherever else the OS creates temp directories), and then add the package to that.

I guess the specifics like which env gets activated can be discussed for this option (eg in a github issue), so that the most convenient defaults are chosen.

3 Likes
19

‘y’ and ‘n’ are easy to guess, but ‘o’ and ‘t’ are not, this might be why I did not know about the cool feature that ‘o’ is. It would be nice if this was more obvious, either with a link to the specific page in the documentation or with a bit more description in the text.

Would the following make things more clear?

julia> using Revise
 │ Package Revise not found, but a package named Revise is available from a registry. 
 │ Install package?
 │   (MyEnvironment) pkg> add Revise 
 └ (y)es / (n)o / (o)ther options [y]:

Instead of using parenthesis around the first letter, it could also be coloured.

7 Likes
20

I think the idea here is that one loads packages from either the current environment, or from paths that are below the currently activated environment in the load path. If the temporary environment is above the global main environment, then activating the global environment while loading packages from the temporary would violate this.

As an example, if we look at the case where the temp environment is below the global main, we find that adding a package leaves the activated environment unchanged:

julia> push!(Base.LOAD_PATH, mktempdir())
4-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"
 "/tmp/jl_sSfw1q"

julia> using FillArrays
 │ Package FillArrays not found, but a package named FillArrays is available from a registry. 
 │ Install package?
 │   (@v1.9) pkg> add FillArrays 
 └ Select environment:
   1: `~/.julia/environments/v1.9/Project.toml` (@v#.#)
 > 2: `/tmp/jl_sSfw1q` (/tmp/jl_sSfw1q)
   Resolving package versions...
    Updating `/tmp/jl_sSfw1q/Project.toml`
  [1a297f60] + FillArrays v1.6.1
    Updating `/tmp/jl_sSfw1q/Manifest.toml`
  [1a297f60] + FillArrays v1.6.1
  [56f22d72] + Artifacts
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [e66e0078] + CompilerSupportLibraries_jll v1.0.5+0
  [4536629a] + OpenBLAS_jll v0.3.21+4
  [8e850b90] + libblastrampoline_jll v5.8.0+0

(@v1.9) pkg>

Perhaps instead of insert! one may use push! to ensure that the temp environment is the lowest one, if this behavior is desired.

2 Likes