Experimenting and developing unregistered packages

General Usage
,
1

Hi all, I’m trying to use the following workflow to collaboratively experiment with and develop a package.

The setup is as follows: I have a project directory with Project.toml and a script script.jl that runs an experiment. In Manifest.toml all required package versions are specified, so as to be able to instantiate the environment and run the same experiment, e.g. on a different machine. Among the required packages is PkgA, my own unregistered package, living in whatever git repository, that my friend Alice and I are interested in improving.

So Manifest.toml will have something like

[[PkgA]]
deps = ["PkgB", "PkgC", "PkgD"]
git-tree-sha1 = "abc"
repo-rev = "master"
repo-url = "https://github.com/bob/PkgA.jl"
uuid = "xyz"
version = "0.1.0"

If I just send the project to Alice, she should be able to run the experiment by doing:

shell> cd project
shell> julia
julia> ]
pkg> activate .
(project) pkg> instantiate
[...]
julia> include("script.jl")

Now Alice has an idea: she wants to locally make a change to PkgA, re-run the experiment, and check whether it improves things. My understanding is that she should be doing

julia> ]
(project) pkg> dev PkgA

and then go somewhere maybe in .julia/dev to find the sources, do the change, run the script again. Except that the above instructions yield (Julia 1.0.1):

(project) pkg> dev PkgA
ERROR: The following package names could not be resolved:
 * PkgA (xyz in manifest but not in project)
Please specify by known `name=uuid`.

Am I trying doing it the wrong way?

1 Like
2

This is where you might be off. Exit julia first, then do the dev on where those source files are. What happened here is that you are developing a package from within its own environment. You want to develop it from some other external environment, like v1.0.

1 Like
3

How you do this is key to an efficient workflow.

I would suggest that

  1. both parties dev the package (you need to do this just once),
  2. use git for version control,
  3. “send” the code using pull requests on Github, Gitlab, or similar.

If the project involves multiple such packages, they should all be dev’d.

Committing Manifest.toml should help with finding the exact same versions.

4

If course, I meant to say “send” in very abstract sense: the idea is that of versioning the project containing the experiments scripts and push it to a shared repository.

I think the problem is with dev’ing a non registered package, PkgA in my example. To simplify my question: how is one supposed to dev a non-registered package?

5

Just dev url? Or if you don’t want to put the deved package online, you could dev a relative local path and tell your collaborator to use the same directory structure.

5 Likes
6

Same if I have already git pull’ed the package elsewhere, right? I can dev path (relative or absolute) and it should recognize (through the uuid) that this is the package which was previously instantiated through the manifest, right?

7

Yep, should work.

You could also have the dev’d package as a git submodule of your main package, e.g. in MainPackage/packages/ABC, and then dev packages/ABC after running julia --project=. from MainPackage. That way everything is set up to modify ABCs code for both you and your collaborator, but you can still have a shared canonical version of ABC checked in to the MainPackage repo (identified by a git SHA). Git submodules can be a little tricky to work with if you’re not familiar with them though.

8

Got it. Thanks everyone for the useful insights. BTW, the --project option is not listed when you use --help in Julia 1.0.1. Is that on purpose?

9

Fixed in https://github.com/JuliaLang/julia/pull/29497 and https://github.com/JuliaLang/julia/pull/29731.

2 Likes