I’m working on a package (A) and realized I can actually extract certain type definitions into a separate package (B) and have the original depend on that.
However, I am still in the process of agreeing on an API and may be required in the future to adapt package B.
Additionally, I’m collaborating with somebody else on a research topic and provide them my package via the use of a local registry and a shared group on a private Gitlab instance.
Am I correct in assuming that if I dev both packages A and B, they will always resolve to the most current up-to-date versions? That way I do not need to constantly update the version numbers and the local registry whenever I prototype and work on updates to check on interoperability?
What about running tests on A, would that resolve to the version described in the Manifest.toml or also to the dev’ed version of B?
Yes, you are correct: If you dev-install both packages (with Pkg.develop(path=...)), you get the current up to date versions. Note that if you you do Pkg.develop with just the name, the repos get cloned into the dev-folder in ~/.julia, and don’t automatically update, so be careful with that.
More generally, there are two issues.
First, it is almost impossible to work with unregistered packages that are dependencies fo other (unregistered) packages. The underlying issue is https://github.com/JuliaLang/Pkg.jl/issues/492. I strongly recommend creating a LocalRegistry. It’s surprisingly easy! Any package that should not be (or isn’t ready yet to be) in the General registry should be in your local registry. Once it’s ready for General, you can (and should) delete it from your local registry. Every package should be in one and only one registry.
Second, Pkg also isn’t great at working with the “master”-branch version of dependencies. You have to manually dev-install all the packages in the order of their dependency (in your local testing scripts, CI scripts, etc.). For two packages A and B, this is pretty manageable. Myself, I have this situation with six packages (one of which is not in General), and I’ve found that I had to write some significant tooling. Specifically, a install_org.jl script that gets included in all of the projects’ test-tooling and all of their CI runs. The script dev-installs all packages in the right order, either from a local checkout in a sibling folder or from the master branch on Github.
Thanks for the quick answer, @goerz. Maybe I wasn’t explicit enough but I am already using LocalRegistry. The issue arises when I reason about what would happen when I change the master-branch.
You gave very good advise on that front. I had forgotten but I had already extracted another portion of my main code into its own package and I could quickly test what you talked about.
Indeed, when I dev the modules using their path on disc (in the correct order, as you mentioned), the most up-to-date parts of the code are used and Julia does not rely on the version number in the Manifest. So that’s nice, I can at least partially test extracting the other part of my main package as well.
However, I also tested to see what happens when I actually ]test the currently active (main) package. In that case, Julia automatically grabs the recent version as defined by the Manifest (in the /test folder) and relies solely on the local registry I created. So whatever changes I may make to a new package I extract from my main package, the automatic tests will fail.
Is there a workaround such that even testing uses the modifed deved module?
I know that when I use a package not added to the test manifest that Julia throws a warning about that but runs anyway. Does the test then use the deved version? Or rather, the most up-to-date version (including dev) Julia knows globally about? Then at least for a new package*, you could rapidly test code extraction using Julia’s automatic test system. Edit: Maybe I remembered old behaviour but Julia 1.7.0 errors on test if a package isn’t added properly. So that’s not an option.
*I try to be careful with my wording here. “New package” means extracting a portion of a main package into its own package.
This is why if you have a package A that depends on B, running Pkt.test() for A will use the version of B in the registry, instead of a local checkout of B as you want.
You basically have two choices:
Put code in the runtests.jl file of package A that dev-installs B
Don’t use Pkg.test()
I prefer the second solution. Put your test dependencies in test/Project.toml, and you can run
julia -p test
julia> Pkg.develop(path=path_to_package_B)
julia> Pkg.instantiate()
julia> include("test/runtests.jl")
If you want coverage information, bound checking, etc., you have to start julia with the appropriate flags.
If you want to be fancy (and need to do this across multiple projects) you can write your own version of Pkg.test that uses an existing environment (the test/Project.toml in A) instead of the sandboxed temporary environment it normally uses.
For my own usage, I’ve implemented
which run test/runtests.jl in a subprocess and also gives me automatic coverage reports, via the very nice LocalCoverage package
No offense taken, I just wanted to be clear about that because using LocalRegistry is kind of part of the “problem”. I don’t mean to imply that a local registry is in and of itself a problem, not at all, it’s the only reasonable way of how to handle private packages that depend on each other (just as you mentioned). But then you force the system to use versioning which causes conflicts when you are required to develop two packages in tandem and need to test interoperability at an early stage. (At least at first glance.)
Interestingly enough, after mulling over what you said, I tried simply activating a fresh environment and deved both packages. When I run a test on A in that environment, it works perfectly fine. The test will use the most up-to-date version of B without a hick. That way, I can test if I implement the interface between A and B correctly without having to commit to an updated version number.
That pretty much solves my issue (at least at a local level; I haven’t used any automated systems or pipelines and whatnot, so your other information will certainly come in handy then ). I’m unsure why it didn’t work globally but I like using environments to keep everything clean, so that’s a plus in any case. (Maybe I had package A active…)
). I’m unsure why it didn’t work globally but I like using environments to keep everything clean, so that’s a plus in any case. (Maybe I had package