Lately I’ve been looking into CI/CD with gitlab. This involves creating a bunch of setup yamls with
bash commands much like in Travis.
As I set these up I am always irked by having to do
julia -e "import Pkg; Pkg.add(\"PackagName\"); Pkg.dev(\"/path/to/PackageName2\"); Pkg.update(\"OtherPackage\")"
et cetera (it gets really ugly when you have to do
Tools like this are becoming increasingly common, so the need for increasingly elaborate setup scripts is only going to increase. It would be really nice if we had a dedicated command line tool such as
juliapkg add PackageName
juliapkg dev /path/to/PackageName2
juliapkg up OtherPackage
I think this has come up before, but I didn’t see anything concrete. It seems likely that everyone will agree that this is wanted, but has any thought been given as to whether it would be part of
Pkg or a completely separate project?
Not a full solution, but the
pkg string macro makes this more pleasant:
julia -e 'using Pkg; pkg"add PackageName"; pkg"dev /path/to/PackageName2"'
and, as a bonus, it doesn’t require any special escaping.
You can even put multiple commands on the same line in the pkg"" string!
julia -e 'using Pkg; pkg"add PackageName; dev /path/to/PackageName2"'
Ah yes, that is significantly better than what I was doing. I still think it would be really nice if we had a CLI like
The REPL mode is for interactive usage of Pkg.
The API mode is for programatic usage.
I don’t see what is ugly with the first example.
Admittedly @swt30’s example is a huge improvement. It just somehow feels sort of hackish to me when I’m doing something that for just about anything other than Julia (though I’m sure people will come up with tons of counter-examples) would be done purely in
bash. I’m imagining people having some elaborate CI script with some
bash scripts with lots of
bash string manipulation that just all gets uglier with
julia -e. Perhaps my imagination is running away with me.
I suppose we’ll get some indication of whether anyone really cares about this from the responses to this thread. Surely if it’s something everyone has been thinking lots of people chime in, but if nobody cares I suspect this will be the very last post.
For CI scripts there is the option to put all operations in a script, and just call it as
- julia setup.jl
julia -e "using Pkg; pkg\"$str\""
does what you want?
$ ./juliapkg status
[6e4b80f9] BenchmarkTools v0.4.0
This. This definitely seems to be the right solution. I think I was looking at this the wrong way: you should never have elaborate setup because you should just have a reasonably well-prepared manifest.
As a side note, the command line option
--project does not appear in
@ExpandingMan, @fredrikekre, could you please compile some blog post or some such explaining for us mortals what and how this works exactly? Looks very useful.
Did you look at the repo @fredrikekre linked above? It is pretty self-explanatory. For each task, a subdirectory contains a
Project.toml and a
Manifest.toml file. That directory is activated with
julia --project, then
Pkg.instantiate just takes care of setting up everything as in the manifest.
--instantiate command line argument would make this even more streamlined.
All @fredrikekre is saying is that you should never have to use
bash (except for the commands he demonstrated) because you should already have your Julia environment built with a
Project.toml or a
Manifest.toml that you built from normal use (either API or REPL) of the package manager during the course of development.
So, for example, suppose I have the Julia package
JuliaApplication which is a “stand-alone” application that’s going to get deployed into a new environment somewhere. In a typical Python example for instance, you’d see a configuration script with something like
sudo pip install pkg1 pkg2 pkg3. That’s probably the wrong solution, however, you should be using
pipenv for something like that which (if I understand it correctly) basically just constructs for you a bash script that does whatever it is you would otherwise have done manually.
In Julia this functionality is built-in, so you should already have the equivalent of your Python
bash script sitting around: it’s just your
Manifest.toml. Your package
JuliaApplication should have one that you created over the course of developing it. So, all you need to do is
julia --project=JuliaApplication -e 'using Pkg; pkg"instantiate"'
and you’re done.
I think one of the things that I got hung up on is that I’m going to have a few packages in a pain-in-the-ass corporate gitlab (of course passwords aren’t good enough for them, they have to use RSA even though the ports to the gitlab are only open to a firewalled corporate network that requires MFA to even get onto… you might think I worked at NORAD, trust me, I don’t), so I’m going to have to figure out how to let Julia clone packages from it, otherwise I’ll have to rely on always having the correct relative paths in my
Manifest.jl (which may still be easier).
One additional point: if
JuliaApplication uses a script as an entry point (for example, if it’s a command line utility) you should probably have
using Pkg; Pkg.activate("..") or some such thing at the top of it, that way the script knows that it should run in the appropriate Julia environment.
I suppose what is being described here is the future, not the present?
At the moment I was in fact advised to delete the
Project.toml file from my package to get it registered, and there apparently shouldn’t be a
Note, that this isn’t about having a “main” project file but a separate one that you use when e.g. generate the docs.
Anyway, regarding deleting Project files for METADATA, it is not something I personally agree with should be done (at all) and the CI tools of METADATA should handle that.
I think the one issue is that there isn’t currently a really clean way to generate a
Project.toml from a
REQUIRE. (I wrote a rough script a while back.) The one catch seems to be that you need to manually put in a bit of the metadata for existing projects such as the UUID. I once had a package where I put in a
Project.toml and inadvertently put in the wrong UUID (though I thought I had checked it). I later had some problems which @kristoffer.carlsson swore were unrelated so I believe him. Regardless, go to
JuliaRegistries/General and make sure you have the correct UUID if you are adding a
Project.toml to an existing registered package.
Regardless, if we are just talking about deplying with gitlab CI or something like that, you could always create a separate environment decoupled from any particular project.
I think the only remaining thing I’m confused about in regard to
version. I take it that is expected to match the latest tag? What if we tag but forget to put the version number in
Project.toml, will the registry freak out?
The version is only used during resolution when you have the package in a “checked out” state, e.g.
add Package#branch. It will also be used in the future when packages are tagged (the version in the Project is what will be tagged).