# Current best practices for depencency management for registered packages

**URL:** <https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376>\
**Category:** Tooling\
**Tags:** question\
**Created:** [February 2, 2019, 9:33am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376 "2019-02-02T09:33:51Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 2, 2019, 9:33am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/1 "2019-02-02T09:33:51Z")

</div>

Could someone please summarize the current recommended practices for `REQUIRE` and `Project.toml` for packages that intend to support _Julia 1.0 and later_, and are registered via `attobot`?

Specifically,

1. do we still need a `REQUIRE`, or is a `Project.toml` enough?

2. should the version go in the `Project.toml` and be updated (because mismatches now seem to be checked and rejected), or is it OK to have a `Project.toml` without a `version`? (the latter is the easiest and least error-prone).

3. Should the UUID be the one from `Pkg.METADATA_compatible_uuid`?

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [February 2, 2019, 10:36am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/2 "2019-02-02T10:36:19Z")

</div>

1. Yes, no.

2. You don’t need a version, but if you do it needs to match the tag when registering ([METADATA.jl/check-pr.jl at 9864c6b21cb003c29ebfcf4a6575a8d558206ed6 · JuliaLang/METADATA.jl · GitHub](https://github.com/JuliaLang/METADATA.jl/blob/9864c6b21cb003c29ebfcf4a6575a8d558206ed6/.test/check-pr.jl#L182-L187)).

3. Yes. ([https://github.com/JuliaLang/METADATA.jl/blob/9864c6b21cb003c29ebfcf4a6575a8d558206ed6/.test/check-pr.jl#L168-L181](https://github.com/JuliaLang/METADATA.jl/blob/9864c6b21cb003c29ebfcf4a6575a8d558206ed6/.test/check-pr.jl#L168-L181))

---

<div class="post-metadata">

**Author:** ![gcalderone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gcalderone/32/1539_2.png) [@gcalderone](https://discourse.julialang.org/u/gcalderone)\
**Post date:** [February 2, 2019, 1:25pm UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/3 "2019-02-02T13:25:31Z")

</div>

> [@Tamas\_Papp](#):
>
> do we still need a `REQUIRE` , or is a `Project.toml` enough?

> [@fredrikekre](#):
>
> Yes, no.

I confess I’m still a little confused about the package management… Two release versions have passed since the adoption of Pkg3 (v0.7 and 1.0, we are now at 1.1).  
Then why it is still “yes, no”?  
When will it becomes “no, yes”?  
…and when the time arrives, what would happen to all those registered projects which doesn’t have a Project.toml?

This is of course not to complain about possible delays, all the work you made is definitely awesome!  
It’s just to understand the rationale behind switching to Pkg3 so early in Julia history.

---

<div class="post-metadata">

**Author:** ![00vareladavid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/00vareladavid/32/23521_2.png) [@00vareladavid](https://discourse.julialang.org/u/00vareladavid)\
**Post date:** [February 3, 2019, 12:22am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/4 "2019-02-03T00:22:37Z")

</div>

## Overview

Source code handling has three components: the code loader, package manager, and registration system.

The code loader is built into the language. On `import`, the code loader will look for code in certain locations on the filesystem.

`Pkg`, the package manager, is a stdlib. `Pkg` is responsible for setting up the filesystem so that the code loader can easily find code. If a package is not installed: `Pkg` will determine where that package lives on the network, download/decompress the package, and finally put the code in a location where the code loader can find it.

`Pkg` determines the URL for a package by querying a registry. A registry is simply a database of package metadata.

## Package registration

The current registry system is still based on `METADATA`. `METADATA` only understands `REQUIRE` files. This is why you currently need a `REQUIRE` file to register a package.

`General` is based on the registry system that will replace `METADATA`. `Pkg` understands `General`, but not `METADATA`.

A high level view of the current registration system:

```julia-auto
REQUIRE -> METADATA -> General <- Pkg

```

## New environments system

`Project.toml` represents a system of organizing code based on environments. Recall that the source code handling system consists of three loosely coupled components. An environments-based system is compelling enough that it makes sense to upgrade components as soon as possible. The code loader and package manager have already been upgraded to use this new system.

## Tradeoffs

The current setup means that package consumers can enjoy the benefits of an environment-based system today. The code loader and package manager are already quite stable and have been improving based on user feedback.

The price to pay for these benefits is having to register your package with a `REQUIRE` file. The fact that the current registration system is cumbersome is even more reason to wait for a better designed replacement system.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 4, 2019, 7:59am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/5 "2019-02-04T07:59:00Z")

</div>

> [@00vareladavid](#):
>
> The price to pay for these benefits is having to register your package with a `REQUIRE` file.

Note that I wasn’t complaining about anything, and I understand that things are in transition. I just needed a clarification of about the current status for practical purposes.

---

<div class="post-metadata">

**Author:** ![00vareladavid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/00vareladavid/32/23521_2.png) [@00vareladavid](https://discourse.julialang.org/u/00vareladavid)\
**Post date:** [February 4, 2019, 8:33am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/6 "2019-02-04T08:33:53Z")

</div>

Yes, I know 🙂. I’ve seen a few people who were confused by the registration process (I certainly was), so I was just taking the opportunity to provide some context.

---

<div class="post-metadata">

**Author:** ![gcalderone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gcalderone/32/1539_2.png) [@gcalderone](https://discourse.julialang.org/u/gcalderone)\
**Post date:** [February 4, 2019, 9:14am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/7 "2019-02-04T09:14:58Z")

</div>

> [@Tamas\_Papp](#):
>
> Note that I wasn’t complaining about anything, and I understand that things are in transition

Me neither, really! I just wanted to understand…

And now I’m even more confused:

> [@00vareladavid](#):
>
> The fact that the current registration system is cumbersome is even more reason to wait for a better designed replacement system.

What is a _“replacement system”_? Why do we need it?

AFAIK the general registry is already working, and there is an automatic procedure copying METADATA info into it. Then why can’t we submit merge requests directly on the registry (rather than on metadata) ?

If the above is not possible, why don’t we do the opposite? i.e. register on the general registry with `Project.toml` and automatically generate a `REQUIRE` file for METADATA?

Sorry for all the questions: I don’t consider myself anymore a newbie, and certainly not an expert. Still, Pkg3 is among the most difficult concept I encountered in Julia.

And what’s more problematic is that new users face Pkg3, and its obscure error messages, way before advanced topics such as parametric types, generated functions, traits, etc… How can we convince people using Julia if even advanced beginners (such as me) don’t understand the basics of registering a package?

Please understand this not as a complain, but as a willing to understand…

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 4, 2019, 9:29am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/8 "2019-02-04T09:29:46Z")

</div>

> [@gcalderone](#):
>
> How can we convince people using Julia if even advanced beginners (such as me) don’t understand the basics of registering a package?

But having read this topic, now you do, don’t you? 😉

I am under the impression that features are in development, and the only way to speed it up is contributing. I am confident that we will get something very nice in the end, and so I am fine with waiting. Workarounds in the meantime are OK.

---

<div class="post-metadata">

**Author:** ![00vareladavid](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/00vareladavid/32/23521_2.png) [@00vareladavid](https://discourse.julialang.org/u/00vareladavid)\
**Post date:** [February 4, 2019, 9:47am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/9 "2019-02-04T09:47:46Z")

</div>

It seems I need to work on my tone 😆. I understand no one was complaining, I was just trying to explain the reasoning for the current state of things.

> What is a _“replacement system”_ ?

> why can’t we submit merge requests directly on the registry (rather than on metadata) ?

The replacement system is exactly the system that will allow us to register directly to `General`.

You might ask yourself why no one just tweaks attobot minimally to get the job done. Presumably this is because the whole ecosystem will depend on the new registration infrastructure once it is up. It is difficult to change behavior once so many people depend on it. It makes sense that it is not being rushed.

* * *

I’m curious, do you find working with `Pkg` confusing? Or is just the registration system which is not clear?

---

<div class="post-metadata">

**Author:** ![gcalderone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gcalderone/32/1539_2.png) [@gcalderone](https://discourse.julialang.org/u/gcalderone)\
**Post date:** [February 4, 2019, 11:06am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/10 "2019-02-04T11:06:24Z")

</div>

> [@00vareladavid](#):
>
> You might ask yourself why no one just tweaks attobot minimally to get the job done.

This is exactly what I wonder about. I understand it may not be straightforward, and operability of the ecosystem has a higher priority. But the very fact that we switched to Pkg3 lead me to think, maybe mistakenly, that such piece should already be in place.

> [@00vareladavid](#):
>
> I’m curious, do you find working with `Pkg` confusing? Or is just the registration system which is not clear?

Well, to create a package, according to the [manual](https://julialang.github.io/Pkg.jl/v1/creating-packages/), you should run through the `generate`, `activate`, `add` commands to properly prepare the `Project.toml` and `Manifest.toml` files. But to register it you should throw away those files and prepare a `REQUIRE` file instead. Am I the only one finding this unintuitive?

Also, with Pkg3 it is possible to declare a dependency on an unregistered package, but AFAIK there is no way to declare such dependency in `REQUIRE`. So I can’t deploy a registered package depending on an unregistered one.

As already noted no one here is complaining, and i believe this discussion already clarified the underlying problem, namely that we miss a _“replacement system”_, i.e. a

> [@00vareladavid](#):
>
> system that will allow us to register directly to `General` .

My very humble opinion and proposal, is that these facts, and the natural consequence that we can not take full advantage of all Pkg3 functionalities for registered packages, should be clearly stated somewhere in an official documentation, rather than being scattered in several posts on the Discourse.

Besides, if there was such entry in an official documentation, @Tamas_Papp would not have started this discussion, and I would have been aware of the missing _“replacement system”_ problem.

Sorry for cluttering this discussion with questions due to my ignorance, hopefully this would be useful to someone else.😉

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [February 4, 2019, 11:17am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/11 "2019-02-04T11:17:28Z")

</div>

> [@gcalderone](#):
>
> Also, with Pkg3 it is possible to declare a dependency on an unregistered package

It’s actually not possible.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 4, 2019, 11:18am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/12 "2019-02-04T11:18:51Z")

</div>

> [@gcalderone](#):
>
> But to register it you should throw away those files and prepare a `REQUIRE` file instead. Am I the only one finding this unintuitive?

Not at all. See eg [this issue](https://github.com/JuliaLang/METADATA.jl/issues/19416); it is unfortunate that it was closed without a solution.

Also, you may be interested in

> <https://github.com/JuliaLang/Pkg.jl/issues/849>
>
> Before we can move from METADATA being the source of truth for registered packag…es to JuliaRegistries/General being the source of truth, we need a process and infrastructure for registering packages. Here's an outline of some of my thoughts on what the process should look like.
> 
> \### 1. Request
> 
> \*\*Who: package maintainer\*\*
> 
> Package maintainer proposes a new version tag
> \- perhaps via an API endpoint?
> 
> Possible data to include:
> \- uuid
> \- repo url
> \- branch
> \- tree hash
> \- commit hash
> \- version number
> 
> Or maybe just one of \`patch\`, \`minor\` or \`major\` and version number is automatic based on existing version numbers. This is almost certainly too much data but I wanted to mention all the possible info one might want. Ideally we'd like to automate as much of this as possible from the repo.
> 
> \### 2. Check
> 
> \*\*Who: automated\*\*
> 
> Automated validation of whether a proposed version tag is acceptable.
> 
> Check that:
> \- version number is sensible (next patch, minor or major)
> \- install the package version by itself
> \- run the tests for the installed package
> \- test variations on dependencies
> \- check if compat claims are plausible
> \- reverse dependency testing
> - check semver compliance?
> 
> Produces report of results
> 
> \### 3. Review
> 
> \*\*Who:\*\*
> \- \*\*package maintainer\*\* (always)
> \- \*\*registry manager\*\* (sometimes)
> 
> Maintainer review:
> \- accept or reject based on report
> 
> If maintainer accepts, go to registry manager review:
> \- if meets auto-approval criteria, it is fully approved without manual review
> \- otherwise proposer can request manual review from a registry manager
> - abilities needed:
> - maintainer to give reasoning
> - manager to give feedback
> - manager to make a decision
> - probably good to take place on a PR somewhere
> 
> \### 4. Tagging
> 
> \*\*Who:\*\*
> \- ideally \*\*automated\*\* via GitHub API
> \- otherwise manually done by \*\*package maintainer\*\*
> 
> Propagate git tag to package repo
> \- should used signed tags signed by some authorized entity
> 
> Can we use the github API to create tags automatically?
> \- https://developer.github.com/v3/git/tags/#create-a-tag-object
> \- perhaps an app that only requires that permission
> 
> What would the workflow be for non-github packages?
> \- even if it’s very manual, we should have one
> \- have a git repo endpoint that we create tags at
> - then document how to pull tags from it?
> 
> The tagged tree also needs to be publicly accessible but I think tagging guarantees that in git. If the tag does not match the version approved in the review step above then the version is not properly tagged and the rest of the process is blocked until the tag is fixed.
> 
> \### 5. Registration
> 
> \*\*Who: automated\*\*
> 
> Once a new version has been approved and tagged, it can finally be registered. This consists of making the appropriate updates to the registry repo. This will be completely automatic.

My impression is that the current setup is considered a collection of temporary measures, so no one feels like documenting it. The best source is [this discussion](https://discourse.julialang.org/t/the-current-metadata-release-process/16672), but it hard to find for newbies. I asked my questions because I noticed some PRs in the related code checking the `Project.toml`, and I was curious.

---

<div class="post-metadata">

**Author:** ![gcalderone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gcalderone/32/1539_2.png) [@gcalderone](https://discourse.julialang.org/u/gcalderone)\
**Post date:** [February 4, 2019, 11:23am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/13 "2019-02-04T11:23:42Z")

</div>

> [@fredrikekre](#):
>
> It’s actually not possible.

Then I don’t understand [this](https://julialang.github.io/Pkg.jl/v1/managing-packages/#Adding-packages-1):

 ![image](https://global.discourse-cdn.com/julialang/original/3X/6/9/69838bf427ec3b89439c37acf4b9159de4ce6ae0.png)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 4, 2019, 11:26am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/14 "2019-02-04T11:26:24Z")

</div>

You can add unregistered packages locally. It just won’t work via the registry, ie you can’t do this reliably in a registered package.

See  
[https://github.com/JuliaLang/Pkg.jl/issues/810](https://github.com/JuliaLang/Pkg.jl/issues/810)

---

<div class="post-metadata">

**Author:** ![gcalderone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gcalderone/32/1539_2.png) [@gcalderone](https://discourse.julialang.org/u/gcalderone)\
**Post date:** [February 4, 2019, 11:36am UTC](https://discourse.julialang.org/t/current-best-practices-for-depencency-management-for-registered-packages/20376/15 "2019-02-04T11:36:00Z")

</div>

I thought adding an unregistered package would add something to `Manifest.toml`, which may then be depoyed with the package…

Thanks for pointing out the relevant issue!