I’m trying to develop some local, interdependent, unregistered packages on Julia v1.9. As far as I can tell, it is still really hard to make it work. Am I doing something wrong? Is there a usable workflow on Julia v1.9? I guess I can try LocalRegistry.jl, but I was hoping I wouldn’t need that.
My dependency graph looks like this:
B ← A → C
where arrows point from dependency to dependent. So, B and C both depend on A but do not depend on each other. However, B is a test dependency for C. Is there a simple workflow using either dev or add <github url> that will get all the packages working, including the unit testing of package C, which depends on B?
I was just using add <github url>, which seemed to work fine until I tried to get testing for C to work, where the C unit tests depend on B. So, that seems to be the crux of the issue right now.
I second this. I generally use dev rather than add. If I’m not mistaken, I believe dev integrates better with Revise, so that way you can just change some of your code in B and get it immediately reflected on your current environment. When using add I think you need to commit your changes first, but I’m not quite sure right now. This is definitively not an issue if B doesn’t change often.
Remember you can also use relative paths… so dev ../B is also an option.
The only way I’ve found to make this work without jumping through crazy hoops is to register those packages in a LocalRegistry.
As a general rule, any package that is a dependency (even a test dependency) for any other package should be registered in some registry. Luckily, working with LocalRegistry is really straightforward.
It doesn’t seem to work. If I do that (dev B and move it to extras and targets) and then run test on the package C, I get the following error:
ERROR: expected package `B [1f83ef90]` to be registered
Furthermore, if I run status, I get this warning:
Warning The project dependencies or compat requirements
have changed since the manifest was last resolved. It is
recommended to `Pkg.resolve()` or consider `Pkg.update()`
if necessary.
Then if I run resolve or update the entry for the dev’d B package gets removed from the manifest file:
(C) pkg> update
Updating registry at `~/.julia/registries/General.toml`
No Changes to `~/projects/C/Project.toml`
Updating `~/projects/C/Manifest.toml`
[1f83ef90] - B v0.1.0 `~/projects/B`
[56ddb016] - Logging
[8dfed614] - Test
Once the dev’d B entry is removed from the manifest, there is no hope that Julia will be able to properly run the tests for package C.
I also tried using Project.toml/Manifest.toml files in the test directory, but I ran into the dreaded ERROR: can not merge projects. (This is not the first time I’ve encountered that error.) Luckily, I think the “can not merge projects” error might be fixed on Julia 1.10.
You’re right, I edited my reply. It might make sense to report this as a Pkg bug; ideally the manifest should not lose track of the path for C if you’ve moved it to [extras].
I have had multiple times now where I have had a cluster of interconnected, non-registered packages that I want to be able to work on, and I want my collaborators to be able to work on.
I really want to be able to just send them a github link and have them dev that package and have it automatically dev that whole local constellation of packages. The current system is a pretty bad user experience where my collaborator needs to go into a bunch of sub-directories and individually dev them in a specific order to satisfy the dependency-graph.
And before someone brings it up, no I specifically do not want to set up a local registry because I want me and my collaborators to be able to easily edit and update these packages without constantly registering new versions.
Note that registering in a LocalRegistry is much more straightforward and lightweight compared to General: no waiting, no restrictions, just run register() and the new version immediately becomes available to everyone. This doesn’t seem a significant overhead, given the benefits of registering and versioning a package.
If both you and collaborators always need to dev/modify the same multiple packages at once, maybe they shouldn’t really be separate packages?
But yeah more generally I can see what you mean, and what can be simplified wrt current situation…
It means though you can’t really do a Revise based workflow right? Constantly hitting register() as I’m interactively working on a package is a step down in the user experience.
In my opinion this is just something Pkg currently handles poorly. It’s not a terribly big deal or anything, but it’s certainly annoying, and I’ve still never seen a satisfying solution other than just individually deving each package in the correct order.
Yes, but I don’t always want a mono-repo. TTFX has improved a lot, but some of the codebases I’ve shared with collaborators still have annoying long startup times, often for functionality that’s not often needed, so I usually prefer packages (especially because precompilation is now slower than before).
The way Pkg is currently designed though really does push me towards monorepos for this stuff which is not ideal. Again though, not the end of the world, just something that I hope improves.
Monorepos are orthogonal to packages, no? What you check out != how Pkg sees your code base. The key thing is that with a monorepo you can update all them them with a single commit.
I’d still register the packages in a LocalRegistry, but then on top of that write yourself an initialization script that dev-installs all the packages you want to develop together into a single environment. For some inspiration, see the Development notes for the QuantumControl organization and the installorg.jl script that I’m using for the dev environment. I agree, though: Julia does not have good built-in tooling for this sort of thing.
The easiest way that works for me is to modify LOAD_PATH: push!(LOAD_PATH, path_to_local_modules), with all local modules in a subdirectory of path_to_local_modules.
This doesn’t require that modules are “packages”; no requirement for Project.toml, Manifest.toml.
Although this is the simplest and just works, it is not recommended and misses out on benefits of more structured package environments.
I have tried dev path_to_module, but this is repetitive, not easily updated and requires one to navigate a sometimes tricky dependency order.
Somewhat naively, I was expecting or hoping to be able to specify once where each local module is located, and then to be able to add that module to the environment of any other local module.
One way I expected this to work was to make use of stacked environments. dev each local module into say LocalModules/Project.toml, which would act as a kind of local registry.
With this “local registry” environment loaded into the current environment stack, one could just add LocalModuleX to the environment of any other LocalModuleY.
To me, this problem is one of my biggest pain point with Julia.
I wish this problem appeared in the list of biggest gripes in the annual julia survey but it’s not so I cannot tick it.
It is very hard to do properly, using a combination of packages, dev command and local registry.
For an experienced julia developper this can be challenging, for a beginner, it’s virtually impossible to find out how to do this correctly.
Especially since the easiest solution using push!(LOAD_PATH, path_to_local_modules) is systematically discouraged.
I think this problem could be alleviated if issue 4600 got resolved, but not sure. I think this issue is more related with the behaviour of include.
I always wondered if it would be possible to implement dev path/to/local_module where local_module is just a module without being a package. This to me would be a good replacement for the push!(LOAD_PATH, path_to_local_modules) solution.
I agree that this is difficult to get working without a LocalRegistry. I have had luck with the following approach to get local A and B availalble in C’s tests.
Use a test/Project.toml and test/Manifest.toml (test-specific environment) instead of the targets way of adding test-specific dependencies.
dev /path/to/A and dev /path/to/B in C’s test environment
Oh, man. I’m currently struggling with the same: trying to come up with a monorepo layout that’s convenient to work with, can support multiple developers, and doesn’t require jumping through a lot of hoops.
Please note that there is a concept of package directories environment, whereby adding a directory to the LOAD_PATH, you can load packages directly from subdirectories of the main package dir. But it doesn’t really work as advertised, mainly, IMHO, because the package manager doesn’t seem to be aware of this concept at all. In particular:
it can’t add packages from this environment to your active project;
it can’t resolve external dependencies (from the General registry) in the packages from this environment;
once you add a package A to the dependency of package B within the environment, it can’t update its Project.toml interactively anymore, because it will always fail to identify A.
I wish such package directories envrionment were handled more similarly to the stdlib (which is also just a monorepo of interdependant packages, in a way).