How to activate a package in a script?

I admit that the following problem might arise only because my team is mixing two concepts–Package and Application–in a single Project. Still, let me pose the problem:

I have a package “Pope” that serves mainly as a library of analysis tools (for superconducting x-ray microcalorimeter data) but also includes a few scripts to be run stand-alone, say, scripts/quick_look_analysis.jl. Suppose this script begins like this:

#!/usr/bin/env julia

using ArgParse
# Here follows code that parses command line arguments, in order to fail as quickly as possible.
# And now we do a "using" on the other needed packages
using Pope
# Then the code that uses the arguments and does physics.
Pope.dostuff()

Now, if I run the above, assuming I have not done a Pkg.add("ArgParse") in my default Julia 1.x environment, then the using ArgParse line will fail.

Here, on the other hand, is a 2-line fix that seems to work. It assumes that I can activate the package in which the script lives and thereby get access to ArgParse.

#!/usr/bin/env julia

using Pkg
Pkg.activate(normpath(joinpath(@__DIR__, "..")))
using ArgParse
# Here follows code that parses command line arguments, in order to fail as quickly as possible.
# And now we do a "using" on the other needed packages
using Pope
# Then the code that uses the arguments and does physics.
Pope.dostuff()

Does this seem a sensible way to get access to my package’s dependencies? Is there a better way? Is there a canonical way?

2 Likes

You can just make a regular old temporary directory to put your environment in. One of Pkg’s many strengths is that it separates the set of installed packages from the active environment. This means creating new environments is virtually free!

You can think of Pkg.add as “make sure these packages are available”. If they are already installed somewhere else, Pkg will know and you don’t have to reinstall them.

#!/usr/bin/env julia
import Pkg

tempdir = mktempdir()
Pkg.activate(tempdir)
Pkg.add(["ArgParse", "Pope"])

using ArgParse, Pope
Pope.dostuff()
2 Likes

Both of these solutions print out a lot of unwanted text… for example @00vareladavid’s solution on my computers starts by outputting:

>julijulia oneilg$ julia script.jl
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Updating registry at `~/.julia/registries/QSGRegistry`
  Updating git-repo `https://github.com/ggggggggg/QSGRegistry.jl`
 Resolving package versions...

is there an easy way to suppress this output?

Also I think there is a julia --project flag, but I can’t find documentation for it.

Not on the official binaries yet, but https://github.com/JuliaLang/Pkg.jl/pull/1053 just merged a few days ago. In the near future you will be able to do Pkg.add("Example"; io=io) to capture all Pkg output.

2 Likes

I notice you’re the author of the PR, so thanks. I can’t tell from the PR, will Pkg.activate("ABC"; io=io) work as well?

Sure! It should work for all Pkg API commands.

1 Like

Thanks for the tips on the behavior of new Pkg. It has been difficult for me to make sense of the many changes, particularly the changes in philosophy, but I am getting there.

I like your temporary directory solution, and I will use it with or without the extra output. I’ll also look forward to the Pkg.func("name"; io=io) feature.

One extra note: @00vareladavid didn’t know this, butPope.jl happens to be a “local project”: it’s not in the main Julia Registry. That means that the tempdir solution will not work in this specific case, because Pkg doesn’t know how to find my project. We would need to add our local registry with Pkg.Registry.add() for a complete solution (Julia 1.1 or later).

What to do for local projects like this? I see at least three solutions, but I only know how to make the first two work:

  1. You could just accept that you are the kind of person who runs scripts with certain package dependencies (here, ArgParse and Pope), and you can add them to your default environment with Pkg.add() or Pkg.dev(). That one-time step should make this problem go away.
  2. If don’t want to rely on everything being in the default environment, and if you are actively developing your local project and want to run the latest version of the script, then I’d suggest the solution I proposed in the original post: Pkg.activate(normpath(joinpath(@__DIR__, "..")))
  3. If you have a local registry, and you are willing to use the latest registered version instead of some local working copy, then the solution is going to require some registry wizardry. I do not have the foo I need to make this work. (And anyway, that really should be a separate discussion.)

On point 3 (custom registry), see Creating a custom registry. One of the contributors has worked out a step by step guide.

Thanks @hendri54. And I should point out that @ggggggggg is my Julia-using partner. He used this very guide (and your help here on Discourse) to get our private registry going. I think he and I could make point 3 work. Or at least, he could.

If choosing between idea 1 and 2, I guess the key question is whether you plan to use the script only on your own machine or you need to distribute it to a bunch of computers in various labs. Idea 1 is easiest and works the fastest at run-time, provided that you’re working at the same REPL day after day. Idea 2 is probably better if you need to set up the script to work immediately on multiple computers.

I usually have a script header like this

#!/bin/bash
#=
JULIA="${JULIA:-julia}"
JULIA_CMD="${JULIA_CMD:-$JULIA --color=yes --startup-file=no}"
export JULIA_PROJECT="$(dirname ${BASH_SOURCE[0]})"
export JULIA_LOAD_PATH=@:@stdlib  # exclude default environment
exec $JULIA_CMD -e 'include(popfirst!(ARGS))' "${BASH_SOURCE[0]}" "$@"
=#

using MyPackage
...

It’s equivalent to Pkg.activate but:

  • Setting up JULIA_PROJECT outside Julia makes it easy to invoke the script in different ways (e.g., using another project)
  • JULIA_LOAD_PATH=@:@stdlib is important if you want to avoid accidentally relying on your default environment.
1 Like

You might even consider leaving out stdlib here.

2 Likes

Ah, I see.

In the case where the package is unregistered and you want to distribute a single script file, you can just add by URL directly:

tempdir = mktempdir()
Pkg.activate(tempdir) # activate a temporary directory
Pkg.add("ArgParse")
Pkg.add(Pkg.PackageSpec(url="https://github.com/usnistgov/Pope.jl")) # add by URL directly

If you want to distribute the whole directory, then I would agree with your option 2.

The tricky bit of that last proposal is to ensure that the right version of Pope.jl is used.
The first time this is run for a given Julia installation, the Pkg.add will download the package from github. But when you run the same script for the second time, I don’t think one can guarantee that the package is re-downloaded and recompiled.
Two approaches that seem to work:

  1. rsync the code to the remote machine and dev it everywhere.
  2. Create your own registry and Pkg.update to the latest version (or Pkg.add a fixed version, if desired).

If there is an easier way, I would love to learn it.

Why not use Manifest.toml? You don’t even need to create registries for non-registered packages.

In my experience, I cannot get Julia to update unregistered packages (unless they are dev’d of course). That would be on the remote computer.

The workflow that I have in mind is: Edit on the local computer. Upload main package and unregistered dependencies to the remote (using rsync). Launch computations on the remote via a script that ensures (somehow) that the most recently uploaded versions of all dependencies are used.

Again, there may well be a way to do all this without a registry that I don’t know about.

Pkg.update() updates all packages that are not deved so that should work just fine.

My comment referred to

There, the package is unregistered and simply using add on the remote would not ensure that the latest uploaded code is compiled. And update would ignore the unregistered dependency.

Hence, I either need to register all the dependencies (and maintain a private registry); then I can update.
Or I need to dev the dependencies on the remote as well.

Unless there is a simpler way that I am unaware of… such as forcing a recompile of certain packages.

Why not?

No, if you have added by URL as David suggested Pkg will update it to latest master, otherwise it is a bug.

3 Likes

Sorry about that misunderstanding. I had convinced myself (by running examples) that add did not always download and recompile the latest master. This simplifies matters quite a bit.