I admit that the following problem might arise only because my team is mixing two concepts–Package and Application–in a single Project. Still, let me pose the problem:
I have a package “Pope” that serves mainly as a library of analysis tools (for superconducting x-ray microcalorimeter data) but also includes a few scripts to be run stand-alone, say, scripts/quick_look_analysis.jl. Suppose this script begins like this:
#!/usr/bin/env julia
using ArgParse
# Here follows code that parses command line arguments, in order to fail as quickly as possible.
# And now we do a "using" on the other needed packages
using Pope
# Then the code that uses the arguments and does physics.
Pope.dostuff()
Now, if I run the above, assuming I have not done a Pkg.add("ArgParse") in my default Julia 1.x environment, then the using ArgParse line will fail.
Here, on the other hand, is a 2-line fix that seems to work. It assumes that I can activate the package in which the script lives and thereby get access to ArgParse.
#!/usr/bin/env julia
using Pkg
Pkg.activate(normpath(joinpath(@__DIR__, "..")))
using ArgParse
# Here follows code that parses command line arguments, in order to fail as quickly as possible.
# And now we do a "using" on the other needed packages
using Pope
# Then the code that uses the arguments and does physics.
Pope.dostuff()
Does this seem a sensible way to get access to my package’s dependencies? Is there a better way? Is there a canonical way?
You can just make a regular old temporary directory to put your environment in. One of Pkg’s many strengths is that it separates the set of installed packages from the active environment. This means creating new environments is virtually free!
You can think of Pkg.add as “make sure these packages are available”. If they are already installed somewhere else, Pkg will know and you don’t have to reinstall them.
#!/usr/bin/env julia
import Pkg
tempdir = mktempdir()
Pkg.activate(tempdir)
Pkg.add(["ArgParse", "Pope"])
using ArgParse, Pope
Pope.dostuff()
Not on the official binaries yet, but https://github.com/JuliaLang/Pkg.jl/pull/1053 just merged a few days ago. In the near future you will be able to do Pkg.add("Example"; io=io) to capture all Pkg output.
Thanks for the tips on the behavior of new Pkg. It has been difficult for me to make sense of the many changes, particularly the changes in philosophy, but I am getting there.
I like your temporary directory solution, and I will use it with or without the extra output. I’ll also look forward to the Pkg.func("name"; io=io) feature.
One extra note: @00vareladavid didn’t know this, butPope.jl happens to be a “local project”: it’s not in the main Julia Registry. That means that the tempdir solution will not work in this specific case, because Pkg doesn’t know how to find my project. We would need to add our local registry with Pkg.Registry.add() for a complete solution (Julia 1.1 or later).
What to do for local projects like this? I see at least three solutions, but I only know how to make the first two work:
You could just accept that you are the kind of person who runs scripts with certain package dependencies (here, ArgParse and Pope), and you can add them to your default environment with Pkg.add() or Pkg.dev(). That one-time step should make this problem go away.
If don’t want to rely on everything being in the default environment, and if you are actively developing your local project and want to run the latest version of the script, then I’d suggest the solution I proposed in the original post: Pkg.activate(normpath(joinpath(@__DIR__, "..")))
If you have a local registry, and you are willing to use the latest registered version instead of some local working copy, then the solution is going to require some registry wizardry. I do not have the foo I need to make this work. (And anyway, that really should be a separate discussion.)
Thanks @hendri54. And I should point out that @ggggggggg is my Julia-using partner. He used this very guide (and your help here on Discourse) to get our private registry going. I think he and I could make point 3 work. Or at least, he could.
If choosing between idea 1 and 2, I guess the key question is whether you plan to use the script only on your own machine or you need to distribute it to a bunch of computers in various labs. Idea 1 is easiest and works the fastest at run-time, provided that you’re working at the same REPL day after day. Idea 2 is probably better if you need to set up the script to work immediately on multiple computers.
The tricky bit of that last proposal is to ensure that the right version of Pope.jl is used.
The first time this is run for a given Julia installation, the Pkg.add will download the package from github. But when you run the same script for the second time, I don’t think one can guarantee that the package is re-downloaded and recompiled.
Two approaches that seem to work:
rsync the code to the remote machine and dev it everywhere.
Create your own registry and Pkg.update to the latest version (or Pkg.add a fixed version, if desired).
If there is an easier way, I would love to learn it.
In my experience, I cannot get Julia to update unregistered packages (unless they are dev’d of course). That would be on the remote computer.
The workflow that I have in mind is: Edit on the local computer. Upload main package and unregistered dependencies to the remote (using rsync). Launch computations on the remote via a script that ensures (somehow) that the most recently uploaded versions of all dependencies are used.
Again, there may well be a way to do all this without a registry that I don’t know about.
There, the package is unregistered and simply using add on the remote would not ensure that the latest uploaded code is compiled. And update would ignore the unregistered dependency.
Hence, I either need to register all the dependencies (and maintain a private registry); then I can update.
Or I need to dev the dependencies on the remote as well.
Unless there is a simpler way that I am unaware of… such as forcing a recompile of certain packages.
Sorry about that misunderstanding. I had convinced myself (by running examples) that add did not always download and recompile the latest master. This simplifies matters quite a bit.