We really need a CLI that hooks directly into Pkg

Internals & Design
1

Lately I’ve been looking into CI/CD with gitlab. This involves creating a bunch of setup yamls with bash commands much like in Travis.

As I set these up I am always irked by having to do

julia -e "import Pkg; Pkg.add(\"PackagName\"); Pkg.dev(\"/path/to/PackageName2\"); Pkg.update(\"OtherPackage\")"

et cetera (it gets really ugly when you have to do pathof).

Tools like this are becoming increasingly common, so the need for increasingly elaborate setup scripts is only going to increase. It would be really nice if we had a dedicated command line tool such as pacman, apt, pip or gem. Perhaps

juliapkg add PackageName
juliapkg dev /path/to/PackageName2
juliapkg up OtherPackage

I think this has come up before, but I didn’t see anything concrete. It seems likely that everyone will agree that this is wanted, but has any thought been given as to whether it would be part of Pkg or a completely separate project?

2 Likes
2

Not a full solution, but the pkg string macro makes this more pleasant:

julia -e 'using Pkg; pkg"add PackageName"; pkg"dev /path/to/PackageName2"'

and, as a bonus, it doesn’t require any special escaping.

1 Like
3

You can even put multiple commands on the same line in the pkg"" string!

julia -e 'using Pkg; pkg"add PackageName; dev /path/to/PackageName2"'
2 Likes
4

Ah yes, that is significantly better than what I was doing. I still think it would be really nice if we had a CLI like pip or gem however.

2 Likes
5

The REPL mode is for interactive usage of Pkg.
The API mode is for programatic usage.

I don’t see what is ugly with the first example.

6

Admittedly @swt30’s example is a huge improvement. It just somehow feels sort of hackish to me when I’m doing something that for just about anything other than Julia (though I’m sure people will come up with tons of counter-examples) would be done purely in bash. I’m imagining people having some elaborate CI script with some bash scripts with lots of bash string manipulation that just all gets uglier with julia -e. Perhaps my imagination is running away with me.

I suppose we’ll get some indication of whether anyone really cares about this from the responses to this thread. Surely if it’s something everyone has been thinking lots of people chime in, but if nobody cares I suspect this will be the very last post.

7

For CI scripts there is the option to put all operations in a script, and just call it as

- julia setup.jl

or similar.

2 Likes
8

Or perhaps even better: Use a custom Project.toml + Manifest.toml and just instantiate that, see e.g. Literate.jl/.travis.yml at 22c6d530d3d331cd0b260c745bd1d48274c56dfd · fredrikekre/Literate.jl · GitHub where I instantiate a custom project to submit coverage and Literate.jl/.travis.yml at 22c6d530d3d331cd0b260c745bd1d48274c56dfd · fredrikekre/Literate.jl · GitHub for another project to build the docs.

6 Likes
9 
#!/bin/bash
str=""$*""
julia -e "using Pkg; pkg\"$str\""

does what you want?

$ ./juliapkg status
    Status `~/.julia/environments/v1.0/Project.toml`
  [6e4b80f9] BenchmarkTools v0.4.0
...
1 Like
10

This. This definitely seems to be the right solution. I think I was looking at this the wrong way: you should never have elaborate setup because you should just have a reasonably well-prepared manifest.

As a side note, the command line option --project does not appear in man julia.

2 Likes
11

@ExpandingMan, @fredrikekre, could you please compile some blog post or some such explaining for us mortals what and how this works exactly? Looks very useful.

12

Did you look at the repo @fredrikekre linked above? It is pretty self-explanatory. For each task, a subdirectory contains a Project.toml and a Manifest.toml file. That directory is activated with julia --project, then Pkg.instantiate just takes care of setting up everything as in the manifest.

Having an --instantiate command line argument would make this even more streamlined.

1 Like
13

All @fredrikekre is saying is that you should never have to use Pkg from bash (except for the commands he demonstrated) because you should already have your Julia environment built with a Project.toml or a Manifest.toml that you built from normal use (either API or REPL) of the package manager during the course of development.

So, for example, suppose I have the Julia package JuliaApplication which is a “stand-alone” application that’s going to get deployed into a new environment somewhere. In a typical Python example for instance, you’d see a configuration script with something like sudo pip install pkg1 pkg2 pkg3. That’s probably the wrong solution, however, you should be using pipenv for something like that which (if I understand it correctly) basically just constructs for you a bash script that does whatever it is you would otherwise have done manually.

In Julia this functionality is built-in, so you should already have the equivalent of your Python pipenv bash script sitting around: it’s just your Manifest.toml. Your package JuliaApplication should have one that you created over the course of developing it. So, all you need to do is

julia --project=JuliaApplication -e 'using Pkg; pkg"instantiate"'

and you’re done.

I think one of the things that I got hung up on is that I’m going to have a few packages in a pain-in-the-ass corporate gitlab (of course passwords aren’t good enough for them, they have to use RSA even though the ports to the gitlab are only open to a firewalled corporate network that requires MFA to even get onto… you might think I worked at NORAD, trust me, I don’t), so I’m going to have to figure out how to let Julia clone packages from it, otherwise I’ll have to rely on always having the correct relative paths in my Manifest.jl (which may still be easier).

One additional point: if JuliaApplication uses a script as an entry point (for example, if it’s a command line utility) you should probably have using Pkg; Pkg.activate("..") or some such thing at the top of it, that way the script knows that it should run in the appropriate Julia environment.

4 Likes
14

I suppose what is being described here is the future, not the present?
At the moment I was in fact advised to delete the Project.toml file from my package to get it registered, and there apparently shouldn’t be a Manifest.toml either…

15

Note, that this isn’t about having a “main” project file but a separate one that you use when e.g. generate the docs.

Anyway, regarding deleting Project files for METADATA, it is not something I personally agree with should be done (at all) and the CI tools of METADATA should handle that.

16

I think the one issue is that there isn’t currently a really clean way to generate a Project.toml from a REQUIRE. (I wrote a rough script a while back.) The one catch seems to be that you need to manually put in a bit of the metadata for existing projects such as the UUID. I once had a package where I put in a Project.toml and inadvertently put in the wrong UUID (though I thought I had checked it). I later had some problems which @kristoffer.carlsson swore were unrelated so I believe him. Regardless, go to JuliaRegistries/General and make sure you have the correct UUID if you are adding a Project.toml to an existing registered package.

Regardless, if we are just talking about deplying with gitlab CI or something like that, you could always create a separate environment decoupled from any particular project.

I think the only remaining thing I’m confused about in regard to Project.toml is version. I take it that is expected to match the latest tag? What if we tag but forget to put the version number in Project.toml, will the registry freak out?

17

The version is only used during resolution when you have the package in a “checked out” state, e.g. dev or add Package#branch. It will also be used in the future when packages are tagged (the version in the Project is what will be tagged).

1 Like