# We really need a CLI that hooks directly into Pkg

**URL:** <https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639>\
**Category:** Internals & Design\
**Created:** [September 6, 2018, 7:42pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639 "2018-09-06T19:42:15Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 6, 2018, 7:42pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/1 "2018-09-06T19:42:15Z")

</div>

Lately I’ve been looking into CI/CD with gitlab. This [involves](https://docs.gitlab.com/ee/ci/quick_start/README.html) creating a bunch of setup yamls with `bash` commands much like in Travis.

As I set these up I am always irked by having to do

```julia
julia -e "import Pkg; Pkg.add(\"PackagName\"); Pkg.dev(\"/path/to/PackageName2\"); Pkg.update(\"OtherPackage\")"

```

et cetera (it gets really ugly when you have to do `pathof`).

Tools like this are becoming increasingly common, so the need for increasingly elaborate setup scripts is only going to increase. It would be really nice if we had a dedicated command line tool such as `pacman`, `apt`, `pip` or `gem`. Perhaps

```julia
juliapkg add PackageName
juliapkg dev /path/to/PackageName2
juliapkg up OtherPackage

```

I think this has come up before, but I didn’t see anything concrete. It seems likely that everyone will agree that this is wanted, but has any thought been given as to whether it would be part of `Pkg` or a completely separate project?

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 6, 2018, 7:55pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/2 "2018-09-06T19:55:06Z")

</div>

Not a full solution, but the `pkg` string macro makes this more pleasant:

```julia
julia -e 'using Pkg; pkg"add PackageName"; pkg"dev /path/to/PackageName2"'

```

and, as a bonus, it doesn’t require any special escaping.

---

<div class="post-metadata">

**Author:** ![swt30](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/swt30/32/4667_2.png) [@swt30](https://discourse.julialang.org/u/swt30)\
**Post date:** [September 6, 2018, 7:57pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/3 "2018-09-06T19:57:58Z")

</div>

You can even put multiple commands on the same line in the pkg"" string!

```bash
julia -e 'using Pkg; pkg"add PackageName; dev /path/to/PackageName2"'

```

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 6, 2018, 8:00pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/4 "2018-09-06T20:00:48Z")

</div>

> [@swt30](#):
>
> You can even put multiple commands on the same line in the pkg"" string!

Ah yes, that is significantly better than what I was doing. I still think it would be really nice if we had a CLI like `pip` or `gem` however.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 6, 2018, 8:09pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/5 "2018-09-06T20:09:30Z")

</div>

The REPL mode is for interactive usage of Pkg.  
The API mode is for programatic usage.

I don’t see what is ugly with the first example.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 6, 2018, 8:19pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/6 "2018-09-06T20:19:10Z")

</div>

> [@kristoffer.carlsson](#):
>
> I don’t see what is ugly with the first example.

Admittedly @swt30’s example is a huge improvement. It just somehow feels sort of hackish to me when I’m doing something that for just about anything other than Julia (though I’m sure people will come up with tons of counter-examples) would be done purely in `bash`. I’m imagining people having some elaborate CI script with some `bash` scripts with lots of `bash` string manipulation that just all gets uglier with `julia -e`. Perhaps my imagination is running away with me.

I suppose we’ll get some indication of whether anyone really cares about this from the responses to this thread. Surely if it’s something everyone has been thinking lots of people chime in, but if nobody cares I suspect this will be the very last post.

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [September 6, 2018, 8:46pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/7 "2018-09-06T20:46:35Z")

</div>

For CI scripts there is the option to put all operations in a script, and just call it as

```julia
- julia setup.jl

```

or similar.

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [September 6, 2018, 8:49pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/8 "2018-09-06T20:49:57Z")

</div>

Or perhaps even better: Use a custom `Project.toml` + `Manifest.toml` and just instantiate that, see e.g. [Literate.jl/.travis.yml at 22c6d530d3d331cd0b260c745bd1d48274c56dfd · fredrikekre/Literate.jl · GitHub](https://github.com/fredrikekre/Literate.jl/blob/22c6d530d3d331cd0b260c745bd1d48274c56dfd/.travis.yml#L16-L18) where I instantiate a custom project to submit coverage and [Literate.jl/.travis.yml at 22c6d530d3d331cd0b260c745bd1d48274c56dfd · fredrikekre/Literate.jl · GitHub](https://github.com/fredrikekre/Literate.jl/blob/22c6d530d3d331cd0b260c745bd1d48274c56dfd/.travis.yml#L25-L26) for another project to build the docs.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [September 6, 2018, 10:54pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/9 "2018-09-06T22:54:25Z")

</div>

```julia
#!/bin/bash
str=""$*""
julia -e "using Pkg; pkg\"$str\""

```

does what you want?

```julia
$ ./juliapkg status
    Status `~/.julia/environments/v1.0/Project.toml`
  [6e4b80f9] BenchmarkTools v0.4.0
...

```

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 6, 2018, 11:10pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/10 "2018-09-06T23:10:23Z")

</div>

> [@fredrikekre](#):
>
> Use a custom `Project.toml` + `Manifest.toml` and just instantiate that

This. This definitely seems to be the right solution. I think I was looking at this the wrong way: you should never have elaborate setup because you should just have a reasonably well-prepared manifest.

As a side note, the command line option `--project` does not appear in `man julia`.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [September 7, 2018, 8:03am UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/11 "2018-09-07T08:03:54Z")

</div>

@ExpandingMan, @fredrikekre, could you please compile some blog post or some such explaining for us mortals what and how this works exactly? Looks very useful.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 7, 2018, 8:23am UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/12 "2018-09-07T08:23:58Z")

</div>

Did you look at the repo @fredrikekre linked above? It is pretty self-explanatory. For each task, a subdirectory contains a `Project.toml` and a `Manifest.toml` file. That directory is activated with `julia --project`, then `Pkg.instantiate` just takes care of setting up everything as in the manifest.

Having an `--instantiate` command line argument would make this even more streamlined.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 7, 2018, 1:47pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/13 "2018-09-07T13:47:25Z")

</div>

> [@yakir12](#):
>
> @ExpandingMan, @fredrikekre, could you please compile some blog post or some such explaining for us mortals what and how this works exactly? Looks very useful.

All @fredrikekre is saying is that you should never have to use `Pkg` from `bash` (except for the commands he demonstrated) because you should already have your Julia environment built with a `Project.toml` or a `Manifest.toml` that you built from normal use (either API or REPL) of the package manager during the course of development.

So, for example, suppose I have the Julia package `JuliaApplication` which is a “stand-alone” application that’s going to get deployed into a new environment somewhere. In a typical Python example for instance, you’d see a configuration script with something like `sudo pip install pkg1 pkg2 pkg3`. That’s probably the wrong solution, however, you should be using `pipenv` for something like that which (if I understand it correctly) basically just constructs for you a bash script that does whatever it is you would otherwise have done manually.

In Julia this functionality is built-in, so you should already have the equivalent of your Python `pipenv` `bash` script sitting around: it’s just your `Manifest.toml`. Your package `JuliaApplication` should have one that you created over the course of developing it. So, all you need to do is

```julia
julia --project=JuliaApplication -e 'using Pkg; pkg"instantiate"'

```

and you’re done.

I think one of the things that I got hung up on is that I’m going to have a few packages in a pain-in-the-ass corporate gitlab (of course passwords aren’t good enough for them, they have to use RSA even though the ports to the gitlab are only open to a firewalled corporate network that requires MFA to even get onto… you might think I worked at NORAD, trust me, I don’t), so I’m going to have to figure out how to let Julia clone packages from it, otherwise I’ll have to rely on always having the correct relative paths in my `Manifest.jl` (which may still be easier).

One additional point: if `JuliaApplication` uses a script as an entry point (for example, if it’s a command line utility) you should probably have `using Pkg; Pkg.activate("..")` or some such thing at the top of it, that way the script knows that it should run in the appropriate Julia environment.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [September 7, 2018, 3:52pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/14 "2018-09-07T15:52:34Z")

</div>

I suppose what is being described here is the future, not the present?  
At the moment I was in fact advised to delete the `Project.toml` file from my package to get it registered, and there apparently shouldn’t be a `Manifest.toml` either…

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 7, 2018, 4:17pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/15 "2018-09-07T16:17:15Z")

</div>

Note, that this isn’t about having a “main” project file but a separate one that you use when e.g. generate the docs.

Anyway, regarding deleting Project files for METADATA, it is not something I personally agree with should be done (at all) and the CI tools of METADATA should handle that.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 7, 2018, 4:34pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/16 "2018-09-07T16:34:25Z")

</div>

I think the one issue is that there isn’t currently a really clean way to generate a `Project.toml` from a `REQUIRE`. (I wrote a rough script a while back.) The one catch seems to be that you need to manually put in a bit of the metadata for existing projects such as the UUID. I once had a package where I put in a `Project.toml` and inadvertently put in the wrong UUID (though I thought I had checked it). I later had some problems which @kristoffer.carlsson swore were unrelated so I believe him. Regardless, go to `JuliaRegistries/General` and make sure you have the correct UUID if you are adding a `Project.toml` to an existing registered package.

Regardless, if we are just talking about deplying with gitlab CI or something like that, you could always create a separate environment decoupled from any particular project.

I think the only remaining thing I’m confused about in regard to `Project.toml` is `version`. I take it that is expected to match the latest tag? What if we tag but forget to put the version number in `Project.toml`, will the registry freak out?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 7, 2018, 5:45pm UTC](https://discourse.julialang.org/t/we-really-need-a-cli-that-hooks-directly-into-pkg/14639/17 "2018-09-07T17:45:33Z")

</div>

> [@ExpandingMan](#):
>
> What if we tag but forget to put the version number in `Project.toml` , will the registry freak out?

The version is only used during resolution when you have the package in a “checked out” state, e.g. `dev` or `add Package#branch`. It will also be used in the future when packages are tagged (the version in the Project is what will be tagged).