Package Manager Documentation

New to Julia
1

Improving package documentation

Today I decided to try and do something constructive to improve the documentation for the Julia Package Manager. As I am very confused by it and a total noob when it comes to package systems, I feel that I am the (at times) uniquely qualified individual for outlining some of the things in the current documentation that might be difficult for new users.

I will lay out my suggestions in a series of posts in this thread. I am working from the current documentation, which I believe to be:

https://julialang.github.io/Pkg.jl/v1/

Note: I’m not giving any answers here if you’ve come looking for info on the package system!

Thanks to those who wrote and documented the package system

Before I begin, I want to express my genuine gratitude to the many individuals who have implemented the package manager and those who have contributed to the documentation. I will be very blunt in my suggestions below, but only because I want to get the business done, not because I don’t appreciate the many individuals who have worked on this.

Glancing through the documentation I see the names kristoffer, frederikekre, stefan, so at least these individuals give away that they should be thanked. I am sure there are many others.

Assumed knowledge of package system user

I will assume that a reader of the documentation uses Julia, that they know they need to install packages in order to do anything beyond the core language in Julia. I will therefore assume that the topic of the documentation is packages and that this is the unit of measure, unless explicitly stated otherwise. Obviously as a developer of a number of packages, I cannot pretend to not know what a package is.

However, I will not assume that the reader is a new user. The documentation has to serve for package developers just as well as for new users.

Note that I have read the entire documentation through once before making any comments here.

Abbreviations

Hopefully the notation I use below will be self-explanatory, but I will use a number of headings and abbreviations, which I now define:

[QUES] A question that a reader might have when reading the documentation. I will restrict myself to questions that are germane to the topic at hand and will try to avoid asking gratuitous questions under this heading. It might be useful to collate answers to these questions in this thread AFTER I have reached section 6 in the documentation. A question will generally imply that something should be added to the documentation at this point.

[OTOP] Off topic question. This is a question which is not germane to the topic at hand, and I don’t expect something to be added to the package documentation, however it might be nice to understand. A reply to this thread would be helpful.

[SUGG] Suggestion. This is a suggestion. People may have varying opinions about whether it is a helpful suggestion. I don’t expect that all my suggestions will be adopted.

[TYPO] Typo. If there is more than one typo in a sentence that I quote, I will include a number in parentheses to indicate how many typos appear in the quote. In most cases the typos are obvious and I will not explain them, but feel free to ask if any of the typos listed are not clear.

[NOTE] Note. This is a comment that might not lead to a specific suggestion, but may explain why I am being overly pedantic at some point, or might illustrate a point I have been making with examples. I will no doubt illustrate my own ignorance in making such comments. But the objective here is not to “clear up a misconception that I seem to have”, but to improve the documentation so that I and others will not have misconceptions.

Structure of this post

I will break the posts down by section in the documentation according to the numbering in the current documentation, and by heading within a section. At the end of each section I will give a list of topics that were not covered in that section that I felt should have been covered.

I volunteer

I am happy to volunteer some time next month to do pull requests against the current docs once I have finally understood the package manager and once everyone has agreed on the canonical answers. But I will only do this if necessary. I understand that the current owners may have design goals for their documentation that don’t align with my expectations. This is normal and so I will only try to contribute after a long delay, if the issues weren’t already addressed by someone else.

Additional notes

I have numbered each comment for easy reference. Admins feel free to remove or rearrange comments as needed.

I apologise if this whole thing seems overly formal. I’m not trying to encourage a formal response or extensive additions to the docs. Many very long comments can be addressed with very short, carefully considered changes.

This is my attempt at being constructive, and the current owners of the package system are not exactly in the next office to me, so this is the best way I can come up with to help, given that I am pretty much ignorant myself and not able to talk with them directly.

I may occasionally make jokes. I can’t help it. It’s a medical condition.

I hope that the focus will be on how to improve the docs for everyone, rather than to deal with my own personal misconceptions or to reply to me. If I misunderstood, I guarantee 100 others are likely to as well. The best solution is to suggest improvements to the docs.

21 Likes
2

[PLEASE READ THE FIRST POST THOROUGHLY BEFORE REPLYING]

1. Introduction

There is also a functional API interface, which is preferred when not working interactively.

a. [NOTE] The way this is worded here (the API documentation itself seem to imply the same), this is mainly useful for scripting access to Pkg. As a user of Julia, I presume this is done via Pkg.blah. A quick glance at the API reference seems to confirm this.

b. [SUGG] “API documentation” could be a link.

Background and Design

local to an individual project

c. [QUES] What is a project? This is not previously defined and I genuinely don’t know what a project is.

The exact set of packages and versions

d. [QUES] What is a version? I assume this means the versions of the packages used in that environment. If something generic like that is meant, and this doesn’t have a specific technical definition unique to Julia, then perhaps “the specific versions used” might be clearer.

checked into a project repository

e. [NOTE] I am confused by this. I understood that packages had repositories. What is the difference between a project repository and a package repository? Is there a list of projects somewhere? My boss told us that we should switch to using projects as he was under the impression from the documentation that a project allowed a number of packages to be managed in a single git repository. He does not like having to update AbstractAlgebra.jl in this git repository and Nemo.jl in another and so on. He believes projects solve this problem, but we have not figured out how.

significantly improving reproducibility of projects

f. [NOTE] This implies that projects were a concept that predated the new package manager. But I am not familiar with the concept from before Julia 1.0.

g. [NOTE] Reading on I conclude that reproducibility here refers to reproducing a list of dependencies, not reproducibility of builds or reproducibility of bugs. What exactly reproducing a project means remains unclear until I understand what a project is.

In Pkg, since each project maintains its own independent set of package versions, you’ll never have this problem again.

h. [NOTE] From my perspective, I had this problem just this morning (or rather one of our users did, and it became my problem shortly thereafter). I clearly misunderstand what problem is solved. But given that I don’t know what a project is, that likely explains why we are having the problem.

if you check out a project on a new system, you can simply materialize the environment described by its manifest file

i. [NOTE] “check out” implies I check out a branch, as in git. I cannot pretend to be ignorant of the fact that Julia uses git for packages. A new user would probably think “check out” is a process you do in Julia.

j. [NOTE] It’s probably pushing things a little to suggest that I did not find the command “materialize” in the API. :slight_smile: It’s clear this whole section is informal. I’m just going to make the perhaps slightly more constructive comment that this does make it sound like there is some additional step required to “materialize” a known-good set of dependencies for a project after “checking it out”. (I think I understand that this is probably actually true in some cases, but I wonder if it will be confusing to new users and is better left for a later section.)

Since environments are managed and updated independently from each other, “dependency hell” is significantly alleviated in Pkg.

k. [NOTE] I have not so far achieved the benefits promised. I am finding that I have the same dependency hell, but now I don’t know how to instruct our users to fix it. We have developed a whole corpus of incantations and mantras to pass on to our users to help them resolve their dependency conflicts. These range from telling them to run “resolve” which seems to magically make some problems go away, to deleting directories to removing projects to removing their entire Julia installation and trying again, to "dev"ing all their projects, etc.

If you want to use the latest and greatest version of some package in a new project but you’re stuck on an older version in a different project, that’s no problem

l. [NOTE] It’s now clear packages and projects are distinct. Obviously a project is a collection of packages. But it does make it seem that in order to use my packages, a user must make a project on their system and maintain it. Either that or we need to make a project for them and somehow distribute it to them. I am not so hopeful the documentation explains how to do this after reading it through more than once.

since they have separate environments they can just use different versions, which are both installed at the same time in different locations on your system

m. [NOTE] This does sound ideal. We often have the situation where the Hecke.jl package relies on an old version of Nemo, but Singular.jl requires a recent version of Nemo. The fact that you can have each project rely on a different set of packages, even though the versions conflict, is great. But so far it has not worked for us. It seems that every time we get a new user of our packages, many mysterious incantations are required to restore balance in the force. They are relying on us to interpret the packaging scriptures for them as they do not seem to be able to navigate it themselves.

The location of each package version is canonical

n. [QUES] Where are canonical package versions stored on my system? What is meant by them being canonical? I thought the idea of manifests/environments was to decouple from a “canonical version”?

so when environments use the same versions of packages, they can share installations

o. [QUES] I’m not sure if “installations” relates to builds, e.g. C libraries that are installed as part of a package build or whether it’s simply referring to where a given package or different versions thereof are “checked out” on my system. I will assume the latter. However, I assume conversely that packages do not have to share “installations”. For example it is ok for Oscar to depend on Hecke and Nemo and for Hecke and Nemo to both depend on different versions of AbstractAlgebra, right?

approach to local environments

p. [QUES] What is a local environment? We have started using “] dev --local MyPkg” because that seemed to solve some problems we had with dev. But we don’t know what it means.

may be familiar to people who have used Python’s virtualenv or Ruby’s bundler

q. [NOTE] I have not personally used virtualenv or bundler. There is no implication here. It’s just background information. However, I will assume this is not directing people to go and learn about those first.

In addition, Julia environments are “stackable”: you can overlay one environment with another and thereby have access to additional packages outside of the primary environment.

r. [QUES] I don’t even know what to ask. After reading the documentation through I have no idea what this means or how I would get this effect, but it does sound very interesting to us, as we manage many packages which all interact. The next sentence in the documentation also does not provide illumination. Why would I want or need my tools in an environment? Granted I don’t use profilers or debuggers (other than MS DEBUG on 16 bit machines of course :-)).

designed to support federated package registries

s. [QUES] What is a package registry? It’s not previously defined, and I am not sure a new user would not simply confuse them with the environments mentioned above, which seems to be the running theme of the entire advertis…, ahem, section.

t. [SUGG] As this is a high level description that will be read by people paid more than me, perhaps a link to where information on federated registries, or just registries in general, might be helpful here.

Additional Remarks

In summary of the above, many of the key terms are not defined, though I would expect them to be in the introduction. I looked briefly ahead to the glossary, which I missed the first few times I read through the documentation, but the descriptions there are very formal.

One of the main problems I am having is getting an understanding of how packages, projects and environments are concretely realised. At the end of the introduction I only understand that environments have manifests of dependencies on packages for projects. This might seem like enough, but only if you knew what those were, concretely.

In particular, I would need more information on the following:

  • What are the upper and lower bounds on the problems each of those concepts tries to address. A user will always come to the documentation with some problem they want to solve and any ill-defined concept will always sound like it is the solution to their particular problem. If it is precisely explained what a project can do for me, I will not leave the documentation with misconceptions.

  • What are the concrete realisations of each of these concepts? Are they defined by a file, a Git repository, or are they merely notional constructs? Does a project have a realisation in a Julia registry, as a GitHub repository, as a local “check out”, as a global default, etc? How would I recognise that I am looking at a project and where would I expect to find one? If I have no conception of what a Julia project is concretely, I will simply conflate it with any number of other conceptions I might have. The same goes for each of these other concepts. Is a manifest a systemwide, global thing? Does it belong in a project or in a package, etc?

I came to the documentation expecting to read about packages. But I am now far less certain I understand what a package is. It sounds like I want to install projects, not packages in my Julia because they will solve all my problems.

9 Likes
3

I will only comment that this is great and very much appreciated. Thank you, @wbhart :pray:t2:.

10 Likes
4

I find it difficult to address your questions/statements/concerns because of the sheer massive wall of text and the formalism you are exposing here on a simple discussion forum. I am wondering because you put so much work into these two posts and all I got from it is the question, what is a project and what is a package, and that I should be carefull to answer because I have to read thoroughly.

So I would like to suggest, that we start with a simple question, like “what is a project and what is a package” and “how are they distinguished”?.

I don’t start with answering, because I am unsure if you are appreciating a sloppy attempt and I am not in the position to offer a formal definition.

2 Likes
5

[PLEASE READ THE FIRST POST THOROUGHLY BEFORE REPLYING]

2. Getting Started

that Pkg commands will modify

a. [SUGG] A minor point: this is confusing given that we already inferred Pkg.blah was used when not in the special package REPL mode. In fact, I could easily understand a new user coming to this page and issuing Pkg.blah commands in this mode. Perhaps it is just clearer to say “package commands”.

The default environment is based on the current Julia version, (v1.1) in the example above, and is located in ~/.julia/environments/ .

b. [QUES] Until earlier today, it was not clear to me what was meant by the “default environment”. In particular in trying to use environments it was not at all clear to me what would happen after I activated an environment and say, exited the Julia session. What is the lifetime of the “current environment”?

c. [SUGG] Perhaps explain that when you start using a new version of Julia and start adding packages, they are by default added to an environment that is specific to that version of Julia. Explain under what circumstances they should expect that environment to persist between sessions and then explain that they can however create their own custom environments which can be activated (is that the right word?) Then explain the persistence of that environment and reemphasize that subsequent package adds will add packages to that custom environment. That is, assuming all that is correct, of course. Now might also be a time to reiterate the relationship between environments and packages/projects. For, if I am not mistaken, they serve a dual purpose, though I am still unclear on the details myself.

The pkg string macro

d. [NOTE] Would you believe, after numerous readings of the documentation, I only noticed this just now? I have noticed people using the string macro and wondered where they found out about it. I was confused because the Note (which I admit to not reading carefully) is in a section on how to use the special package REPL mode. I was quite confused why I would issue Pkg commands in the special REPL mode using pkg"command". I actually get it now, but boy I wish I had understood this ages ago. I am nearly 100% certain most of my colleagues are still using old Pkg.blah commands from earlier versions of Julia with the impression this entire way of accessing Pkg will be deprecated in some later version of Julia. This is reinforced by the fact that some of the old commands come up with deprecation warnings…

use the rm command

e. [SUGG] It would be useful to note what this will do in terms of removing files from the system. There may be some confusion about removing a package from an environment, which this section seems to be about, and removing a package from the system. How is the latter achieved, relative to the former, and which action does this command perform? I think I get that it will be garbage collected if not referred to from any environment, but someone reaching this page without that prior reading will not know that and may be confused what it means.

specified by names in a registry

f. [TYPO] “specified packages by their names in a registry”

g. [SUGG] I’m kind of tempted to suggest the word “registry” be qualified, e.g. “Julia’s package registry”, since we did not actively select a registry in the examples given so far. But, I realise this could be confusing given that you can apparently have custom registries…

an URL

h. [TYPO] I need another heading [PEDT] for pedantic. But see https://blog.apastyle.org/apastyle/2012/04/using-a-or-an-with-acronyms-and-abbreviations.html
I expect a flame war about how URL is pronounced. :slight_smile:

to update a installed package

i. [TYPO] a → an

j. [SUGG] What does this do? Will it update to the latest version, the latest compatible version? Will it update the dependencies? Will it automatically rebuild?

most use cases for simple package management: adding, updating and removing dependencies

k. [SUGG] I find this confusing when reading this in the context of simple package management, which I assume will be mainly done by new users of Julia. The word “dependencies” here will be confusing, as they will not understand why they would need to manage dependencies of the packages they are “installing” in their Julia. Again, this comes down to the dual/triple use of environments (if I’ve understood correctly), e.g. the “default” environment, custom environments and environments as used by projects.

pkg> develop --local Example

l. [QUES] What does this actually do? In particular, what is “local” all about? It has been the source of endless confusion. We used to do Pkg.dev and it put a checkout of a package in .julia/dev/PkgName. I was very confused by this as it didn’t seem to take into account that there were different versions of Julia coming.

m. [QUES] Conversely, sometimes I might also want a single dev’d version of a package on my system, across all Julia versions. How do I achieve that?

your project directory

n. [QUES] What is my project directory and where is it? So far we have only discussed packages and environments, not projects.

visible to your project

o. [QUES] Only to my project? Or also to other projects? Is there nothing like the old Pkg.dev? Managing multiple Git repos for dev’d projects is proving very difficult for us. For example, by default the pushurl will be for the main repo, not for our own personal repo. And I keep forgetting whether it is the dev version or the version in environments I should be editing and which ones are set up to push to my GitHub account, etc. I frequently find myself editing tests in the wrong version…

Now you are back to using the version of Example in the registry.

p. [OTOP] I kind of expected it would go back to whatever state it was in before dev’ing it, e.g. what if I had checked out a custom version?

Additional Remarks

My main concern in this section is being unsure what the results of any action will be. Obviously a “quickstart” is not the place to go into excessive detail, but the core concepts and the likely results of user actions should be clear. As I have noted, there also seems to be some context missing here or there.

I was surprised to not learn about the following in the quickstart:

  • How to share an environment with someone else
  • How to switch environments

[SUGG] The quickstart could possibly benefit from a brief sentence at the start or end stating that information on how to create, develop and register packages is coming in the next section. It’s a minor thing, but the quickstart is clearly aimed at getting Julia users started, not necessarily package developers.

An alternative would be to simply state at the beginning of the quickstart what you will learn, i.e. the scope of the quickstart, so that people who want to learn how to do package development can skip to the information they are looking for, without reading this page. “Getting Started” doesn’t describe the scope of the information.

[SUGG] Perhaps it might be useful to separate out the information on how to work with environments into a separate subsection to clarify the scope of the information for people browsing the page. I don’t have a strong opinion on this though.

1 Like
6

@wbhart said he won’t be reading comments, so you may be speaking into the void until much later here.

2 Likes
7

Accepting the risk of speaking into the void:

I found myself similarly confused about the workings of packages (especially unregistered ones). Not wanting to mess with the official documentation (nor being confident in my ability to do so), I instead wrote up what I found out through experimentation here.

I plan to update the document periodically, partly as documentation for myself. Perhaps it is useful to others. Corrections welcome.

6 Likes
8

I only glanced at this, but like it already.

Regarding the pkg manager, my layman view is that the developers possibly aimed for a very flexible and sophisticated solution which may be difficult to understand (or describe) to new/inexperienced users (like me).

However there are plenty of resources online (such as this forum), that usually help to resolve (my) issues very quickly.

9

[PLEASE READ THE FIRST POST THOROUGHLY BEFORE REPLYING]

3. Managing Packages

Adding registered packages

current project

a. [QUES] Would “current environment” be clearer? Is the default environment considered a project? I see that the next section (4) is about environments, but it seems to exclusively be about creating projects. [I don’t think anyone would notice this unless they were writing this post. Actually, is an environment the manifestation of a package? Honestly, it has not occurred to me until this precise moment to even entertain this thought, and that’s only because I am writing this post.]

Julia’s General registry

b. [TYPO] Looks like a capital letter is missing or there is an extra one.

c. [QUES] Where is Julia’s general registry located? Is there a list somewhere of what is in it? It’s a little hard for a user to know what to add if they don’t have a list on hand.

The status update shows

d. [SUGG] New paragraph?

e. [QUES] What is the “status update”? Or did you mean something like “status function” or “status report” or “current status” or maybe “project status”?

package UUID

f. [QUES] What is a package UUID? I see this as a critical question. How is a UUID generated? By whom? Over what scope is it valid? Does it change when there is a new version of the package? Will it change inside my Julia installation once I “check it out”? Or is it fixed over all of time and space for a given package? Should I as a user of Julia be interested in the UUID for a package, or is it something technical?

The manifest status, in addition, includes the dependencies of explicitly added packages.

g. [QUES] How is it denoting this? I only see one package with a version and one without (obviously because it is a standard package). How do I know which are the dependencies of the explicitly added packages? I was expecting to see some kind of system of arrows to show dependencies. Perhaps a longer example would show this more clearly?

After a package is added to the project

h. [QUES] To the current environment? Or are we really only discussing projects?

after a @ symbol

i. [TYPO] an? At sign - Wikipedia

e.g. @v0.4

j. [QUES] How do I determine what versions are available?

that has not yet included

k. [TYPO]

we will pull updates from that branch

l. [QUES] “Julia will pull updates…” ? Or does this refer to an example that was intended to be added later in the documentation?

m. [SUGG] Git has not been mentioned once, so far, in the documentation. This could be helpful in the introduction.

To go back to tracking the registry version of Example , the command free is used

n. [TYPO] Not sure if you consider the space after Example a typo or not.

o. [QUES] I have noticed that free seems to be used to always go back to the registry version. Isn’t this kinda strange given that someone might have checked out a specific branch, found a problem, decided to dev it and then wants to go back to the registry version of that branch? The way I read it right now, this will not happen. Or am I misunderstanding what free will do?

Adding unregistered packages

it can still be added by instead of the package name giving the URL to the repository to add

p. [SUGG] “it can still be added by providing the repository URL instead of the package name to add”

just like for registered packages

q. [SUGG] “just as we did above for registered packages”

Adding a local package

r. [SUGG] An example would not be wasted. I realise there is a potential problem with Windows vs Linux paths.

when packages are updated

s. [SUGG] when packages are updated.

and the packages updated

t. [TYPO(2)] and the package updated

Developing packages

By only using add your Manifest will always have a “reproducible state”

u. [QUES] Is the branch I checked out also part of that state? I’m not sure if this contradicts previous information on the free command. The implication of this entire paragraph seems to be that if I don’t develop any packages I have a reproducible state. Or are branches also checked out as develop’d packages (or something equivalent)?

Project.toml and Manifest.toml

v. [QUES] We have learned that a manifest is where all the dependencies are stored. But what is Project.toml for?

w. [OTOP] Does the default environment have a Project.toml and Manifest.toml?

load packages at their current state at some path

x. [QUES] Not sure what this means. Do you just mean “at their current state of development”? I assume this means at their master branch?

y. [QUES] Does “at some path” in this sentence belong to “load packages” or “current state”?

Let’s try to dev a registered package

z. [QUES] Can it also be used to dev unregistered packages, e.g. from a Git repo? If not, how do I do this? (It only just occurred to me right this moment after having this problem for some time that I could add it, then dev it. But is this really the intended way to do this?)

When importing Example julia

z1. [TYPO] Julia

z2. [QUES] Is importing supposed to mean something technical? Do you mean adding? Probably not, given the next question… [Edit: oh sorry, I’m dumb. You mean when I do “import MyPkg” or “using MyPkg”. Would importing be better here? Maybe not, as it might imply to some that import is a Pkg command.]

whatever local changes have been made to the files

z3. [QUES] I think this is intentionally written this way, but just want to check that this is in distinction to checking out local packages where just changing the files would not be sufficient.

It is therefore up to you to pull updates, change branches etc.

z4. [QUES] Does Julia offer commands for doing this? Or is it intended that this be done within the actual git repo itself?

the package manager we will simply

z5. [TYPO]

As a general rule, the package manager will never touch files that are tracking a path.

z6. [SUGG] This is almost a duplicate of the same statement a few lines up. I don’t understand why it is being emphasised here (again), if that is intended.

If dev is used on a local path, that path to that package is recorded

z7. [QUES] Recorded where? In the manifest? I don’t completely understand the point that is being made here.

The path will be recorded relative to the project file

z8. [QUES] Wait! So dev is specific to a given project? This doesn’t equate with my experience of using it. [Doh! I think I’m getting confused about the default environment again, which is a project as far as this statement is concerned. So when I’m working on the Singular “project” (not meant in the technical Julia sense) in my default environment, but I dev’d AbstractAlgebra, the dev’d version will be used by Singular if Singular has it as a dependency. This is so terribly confusing when working on a collection of related, but nonetheless independent, packages - a “project” - which isn’t specified by a project file…]

without knowing the exact content of all the packages that are tracking a path

z9. [OTOP] Not an important point, but what does this mean? I had expected it to say “without knowing the contents of all the files at that path”. Why would only knowing the contents of packages tracking the path be sufficient?

if you add a dependency to a package that tracks a local path

z10. [QUES] Wouldn’t a project track a local path, instead of a package? I would expect the package to be at the local path. I unfortunately don’t understand this entire sentence.

z11. [NOTE] This is slightly confusing anyway. At this point I expect from the foregoing that packages don’t have dependencies, but projects do. But then projects can obviously depend on packages which are themselves projects (one presumes). Ours do, anyway. Or are you simply saying that if a package is dev’d and it is itself a project (because it has a Project.toml and Manifest.toml file) that changing either of these files manually in the dev’d location would cause the manifest of the parent project that depends on that package, to be out of sync? Or are you referring to the aforementioned “layering” of environments mentioned in a previous section?

To update sync the Manifest

z12. [QUES] What is an “update sync”. We have been telling users to run resolve for ages now not knowing what it actually does. It seems to fix a remarkable number of problems. But I just now note this is only mentioned in the developing packages section. I’m surprised it fixes issues for some of our users.

Removing packages

Packages can be removed from the current project by

z13. [QUES] Will this also remove dependencies of any package that is removed?

remove all packages that depends

z14. [TYPO] depend

Updating packages

the project is using

z15. [QUES] Not using in the Julia technical sense, but “depends on”, presumably?

the latest compatible version

z16. [NOTE] This is a new concept not previously introduced. It could helpfully be introduced before this point.

all other packages direct dependencies

z17. [TYPO] packages’ ?? or do you mean package

z18. [QUES] I’m not actually sure what is intended by this sentence. Or is it just emphasising that no other packages in that project will be affected. It does seem to imply that dependencies of dependencies might change. But why? Also, what if a package was a direct dependency and an indirect dependency? Might seem like nitpicking, but honestly I’m not really sure what is meant here.

Packages that track a repository are not updated when a minor upgrade is done.

z19. [OTOP] Wow, that’s random!

Pinning a package

A pinned package will never be updated. A package can be pinned using pin as for example

z20. [SUGG] To prevent a package from being updated, it can be pinned. This can be done using pin:

Removing the pin is done using free

z21. [OTOP] Really!? Another use for free? unpin was not considered? I find this odd because every time I read about what free did previously, I assumed that was all it did. But it seems to have a lot of (hidden) consequences! I don’t think most people are going to notice this!

Testing packages

The tests for a package can be run using test command

z22. [TYPO] using the test command

z23. [QUES] What will this actually do? (I actually know, but I’m wondering if a new user would know where the tests for a project are located, or what Julia will look for in a project in order to run its tests.)

z24. [SUGG] Since new package developers will likely be reading this, it might be an opportunity to mention test sets with a link. I had not heard of them until someone asked me why we weren’t using them. I don’t have a strong opinion about including this here though. But if not here, where?

Building a package

The build step of a package is automatically run

z25. [QUES] Again, what explicitly does it do? Where does Julia look for the build script? (I know of course, but the new user would like to find it in the documentation.)

first installed

z26. [QUES] It would be really helpful to have a more precise statement, as we have noticed in the past that packages were being installed multiple times. What about if two dependencies of the same project depend on different versions of the same package? Or even just the same version? Does layering of projects factor in here?

Garbage collecting old, unused packages

Pkg keeps a log of all projects

z27. [QUES/OTOP] Where is the log? Can it be queried?

This is done with the gc command

z28. [QUES] Is it ever done automatically? This is kind of what I think of when I think about a garbage collector.

Preview mode

effects of

z29. [TYPO] effects that

are untouched

z30. [TYPO] “will not be touched” or “would not be altered”

Additional comments

  • The section on developing packages should explain what the dev --local command does.

  • I was really surprised the Developing Packages section did not tell me how to register a package or give the current guidelines on how to do this!

  • I was really surprised that the Developing Packages section did not explain how to issue a new version of a package!

  • I actually anticipated there would be a section on creating a new package on this page. Yes I see that I only have to glance over to the menu on the left to see there is a whole section on creating a package. I still believe a link to there from this page would be helpful. It’s not super obvious that the Managing Packages page wouldn’t contain all this information.

2 Likes
10

[PLEASE READ THE FIRST POST THOROUGHLY BEFORE REPLYING]

4. Working with environments

a. [NOTE] This page seems to be a collection of random bits about projects. I am not sure I would ever click on this page to get answers to my questions. It’s also sandwiched between two different pages that discuss packages. I don’t have a concrete suggestion about how to structure it better.

Creating your own projects

b. [QUES] Wouldn’t it make more sense to discuss how to create a package first?

So far we have added packages to the default project

c. [SUGG] It might be better to mention this before the previous section, rather than after. That might have saved some confusion.

, it is, however

d. [TYPO] New sentence?

independent, projects

e. [TYPO] It’s nitpicking, but this use of a comma does not occur elsewhere in the document.

It should be pointed out if two projects uses

f. [TYPO(n)]

g. [SUGG] The whole sentence seems to be inserted in an otherwise reasonable paragraph. Perhaps it can be moved?

shell> mkdir MyProject

h. [SUGG] At the present moment I don’t recall off the top of my head how to get the shell REPL mode, as I almost never use it. In my opinion, a reminder here really wouldn’t be out of order. I’m sure this can be done by adding 3-4 words and a link in the prior sentence.

changed when the new project is activated

i. [TYPO] “changes when … is activated” or “changed when … was activated”

status command

j. [OTOP] Are status and st both accepted?

and in fact, it has no project or manifest file until we add a package to it

k. [QUES] Really? Then what is the meaning of the last line of the immediately preceding example? It lists Project.toml!

Precompiling a project

You can for example do

l. [TYPO] Not sure if you want a colon here, based on previous usage of colons.

Using someone else’s project

and call

m. [TYPO] Colon?

Simply clone their project

n. [QUES] This is not clear by this point. I expected that all I need to reproduce someone’s project is their Project.toml and Manifest.toml file. So I was expecting instructions on how I reproduce it if I have those. I feel like there is some confusion about the distinction between a package and a project again here. Don’t packages contain files that are not in the Project.toml or Manifest.toml?

o. [QUES] Is this a Julia command, a Pkg command, or something they should do in the shell?

cd to the project directory

p. [SUGG] Add a shell command for changing directory to the example. Or show how to do that in Julia. It’s not super obvious to a new user that Julia somehow keeps track of a current directory.

in the same state

q. [QUES] At the same version/branch/sha? Does “state” include other information? E.g. is some information encoded in the UUID?

it will resolve the latest versions of the dependencies

r. [QUES] Recursively? (My turn to be vague. But I’m not even sure what to ask here.)

Additional comments

  • I vaguely expected this page to tell me how to share my environment with someone else.

  • How do I get my environment onto GitHub so someone else can clone it? Is there a list of these somewhere?

  • The title is “Working with environments” but I don’t see how to switch environments, which environment is the default after I restart my Julia session, whether I need to activate it every time. Perhaps the page title is just misleading me here?

2 Likes
11

[PLEASE READ THE FIRST POST THOROUGHLY BEFORE REPLYING]

5. Creating Packages

a. [SUGG] As previously mentioned, this seems orphaned from the other section on packages.

A package is a project with a

b. [SUGG] I totally missed that word “project” on every previous reading. I suggest splitting that thought into a separate sentence, and including it much earlier in the documentation!

Generating files for a package

To generate files for a new package, use pkg> generate

c. [QUES] This is another major source of confusion for me. Where should I be at this point? What should I have previously done before I start generating a package? Do I need an active project? Do I need to be in a directory? Will it create this new package inside the current environment (I have layering in mind here)? Or is package generation an entirely separate system?

This creates a new project HelloWorld

d. [QUES] Where does it create it?

The Project.toml file contains… and eventual dependencies

e. [TYPO] “and eventually, its dependencies”

f. [QUES] Then what does the manifest contain? I thought that is where a project’s dependencies were recorded?

g. [QUES] Why don’t we have a Manifest.toml at this point?

We can now activate the project

h. [QUES] Fair enough, this has been explained earlier. But if I got here from somewhere else, this is a magic incantation, and I have no idea why I would want to “activate” a project. This is not a hypothetical consideration: I have personally gone in circles for ages wondering whether I need to activate a project before dev’ing it, wondering why I need to dev it if I just activated it, since it is pretty obvious I want to work on it, given that I just created it.

Adding dependencies to the project

i. [NOTE] I have to point out that this section previously confused the hell out of me. I wanted to create a package, but then the documentation immediately goes on to tell me how to add dependencies to the “project”. I just wanted the project to go away, so I could get on with adding files to my package!

j. [NOTE] This section seems to be a repeat of previous information. But I did not previously understand this!!

note how the prompt now shows the name of the newly generated project

k. [SUGG] This should be mentioned in the previous subsection, which it relates to. In this subsection it just confuses the example, which is about adding dependencies, not activation.

The resolver has installed each package with the highest possible version

l. [OTOP] Some documentation about the resolver is eventually going to be needed. Package developers need to understand why it can’t resolve things in any given situation.

that each package enforce

m. [TYPO] enforces

n. [QUES] This is a new concept, introduced here for the first time. It might be better to introduce the concept more fully at an earlier place, or forward link to more information.

Changing src/HelloWorld.jl to

o. [TYPO] Colon?

reloading the package

p. [SUGG] I would be more specific what is meant by reloading (I have activate in mind here).

Adding a build step to the package

by executing the file deps/build.jl

q. [SUGG] By way of introduction, be explicit about why someone might want a build script, and about the fact that one is not necessary for a project. The example fails to illustrate the main use for a build script.

r. [SUGG] It might be a useful place to mention BinaryProvider. I don’t have a strong opinion on this though.

s. [SUGG] Be more specific about where the build script has to go. Some users may think this is a global thing somewhere that is created by some mechanism they are not aware of.

If the build step fails, the output of the build step is printed to the console

t. [TYPO] Colon?

u. [SUGG] Mention the log file explicitly somewhere here, explaining what goes in the log and what goes to the console. I can infer it, but it will be clearer if explicitly stated. Also, I previously totally missed the bit about the log file tacked on to the end of the example.

Adding tests to the package

test/runtests.jl is executed.

v. [SUGG] … in the package is executed.

Test-specific dependencies

This is possible by adding dependencies … to the Project file.

w. [NOTE] I have previously been unsure whether I can manually edit the Project.toml or Manifest.toml file, and to what extent, without messing up Julia internally. Is there a package command to accomplish this?

Here we add the Test standard library as a test-only dependency …

x. [QUES] Is this something that a package developer needs to do? I mean, will the tests run if I don’t do this? If so, why doesn’t Julia do that automatically for me when I generate the package? Also if so, it looks like it belongs in the previous subsection. I wouldn’t look in this subsection if I didn’t believe I had some special package dependencies that are only relevant for my test code.

Package naming guidelines

guidelines applies

y. [TYPO]

but may be useful for other package registries as well

z. [SUGG] I don’t see the usefulness of this comment. If I am putting a private registry behind a corporate firewall, my boss probably has very well-developed notions of how packages should be named that are as distinct from Julia conventions as possible, based on his previous extensive knowledge of SVN, 8 letter filenames and how PDP’s operated.

  • In contrast, JuliaParser provides no new type…

z1. [NOTE] I found this example to be a bit unobvious.

Additional comments

  • Now I really expected to see how to register a package on this page and what the guidelines are for acceptance (or a link to a description of them)

  • A major missing thing on this page is how to create a package based on Julia code from an old version of Julia, or how to turn an existing GitHub repository (which contains Julia code) into a Julia package.

  • Somewhere there needs to be an explanation of how packages relate to modules. Can I have more than one (top level) module in a package? When I add dependencies to a project, does this essentially allow this somehow? Somehow a package seems to be more than a project with a name, UUID and version, even if it is a project. As I understand it, it contains files that define at most one top level module with the same name as the package and any number of submodules. But perhaps I have misunderstood.

  • Also missing is how to create a new version of a package and how to get that recognised by the registry!

1 Like
12

[PLEASE READ THE FIRST POST THOROUGHLY BEFORE REPLYING]

6. Compatibility

a. [SUGG] The title “Restricting compatible versions for dependencies” might be more descriptive. At first glance I assumed this was about compatibility with older Julia versions or the old package manager.

Version specifier format

is given as e.g. 1.2.3 is

b. [TYPO]

More specifically, a version specifier is either given as

c. [SUGG] A version specifier must be either a

d. [SUGG] I would put the previous sentence containing the example after the two sentences about caret specifiers.

Caret specifiers

we have made that choice that

e. [TYPO]

Final Remarks

[QUES/OTOP] How will versioning with respect to Pkg itself be managed. For example, will Pkg3 Project.toml and Manifest.toml files be guaranteed to work with Pkg4, and will Pkg3 ignore sections in Pkg3 files that it doesn’t understand?

As the remaining sections are more technical or serve a different purpose to the main documentation I only intend to report typos, if and when I find time.

I’m now done with what I intended to do today. I hope it will be useful, despite its verbosity!!

I hope the focus will be on providing succinct changes to the existing documentation that can address some of my questions and corrections, rather than on trying to answer me personally. I get that many others implicitly understood the documentation on a first read. Many others like me did not. I feel that improving the docs is the best way to remedy this!

3 Likes
13

Thanks for doing this! I should have composed a response as I read, but overall I think this is really helpful. I’ll read back through and write something up when I’m at a computer, but in the meantime there are a number of threads here in discourse about the project/environment/package distinction (you’re not the only one that’s been confused).

2 Likes
14

Thank you for the feedback!

I’ve created an issue in the Pkg repo to track this discussion: https://github.com/JuliaLang/Pkg.jl/issues/1311

5 Likes