# Developing concurrent, interdependent packages in a local registry

**URL:** <https://discourse.julialang.org/t/developing-concurrent-interdependent-packages-in-a-local-registry/76428>\
**Category:** Package Management\
**Tags:** question\
**Created:** [February 14, 2022, 11:50am UTC](https://discourse.julialang.org/t/developing-concurrent-interdependent-packages-in-a-local-registry/76428 "2022-02-14T11:50:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![doit](https://avatars.discourse-cdn.com/v4/letter/d/9dc877/32.png) [@doit](https://discourse.julialang.org/u/doit)\
**Post date:** [February 14, 2022, 11:50am UTC](https://discourse.julialang.org/t/developing-concurrent-interdependent-packages-in-a-local-registry/76428/1 "2022-02-14T11:50:31Z")

</div>

Hi everyone,

I’m working on a package (A) and realized I can actually extract certain type definitions into a separate package (B) and have the original depend on that.  
However, I am still in the process of agreeing on an API and may be required in the future to adapt package B.  
Additionally, I’m collaborating with somebody else on a research topic and provide them my package via the use of a local registry and a shared group on a private Gitlab instance.

Am I correct in assuming that if I dev both packages A and B, they will always resolve to the most current up-to-date versions? That way I do not need to constantly update the version numbers and the local registry whenever I prototype and work on updates to check on interoperability?  
What about running tests on A, would that resolve to the version described in the Manifest.toml or also to the dev’ed version of B?

Does anybody have experience with that?

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [February 14, 2022, 4:02pm UTC](https://discourse.julialang.org/t/developing-concurrent-interdependent-packages-in-a-local-registry/76428/2 "2022-02-14T16:02:17Z")

</div>

Yes, you are correct: If you dev-install both packages (with `Pkg.develop(path=...)`), you get the current up to date versions. Note that if you you do `Pkg.develop` with just the name, the repos get cloned into the dev-folder in `~/.julia`, and don’t automatically update, so be careful with that.

More generally, there are two issues.

First, it is almost impossible to work with unregistered packages that are dependencies fo other (unregistered) packages. The underlying issue is [https://github.com/JuliaLang/Pkg.jl/issues/492](https://github.com/JuliaLang/Pkg.jl/issues/492). I strongly recommend creating a [LocalRegistry](https://github.com/GunnarFarneback/LocalRegistry.jl). It’s surprisingly easy! Any package that should not be (or isn’t ready yet to be) in the General registry should be in your local registry. Once it’s ready for General, you can (and should) delete it from your local registry. Every package should be in one and only one registry.

Second, `Pkg` also isn’t great at working with the “master”-branch version of dependencies. You have to manually dev-install all the packages in the order of their dependency (in your local testing scripts, CI scripts, etc.). For two packages `A` and `B`, this is pretty manageable. Myself, I have this situation with six packages (one of which is not in General), and I’ve found that I had to write some significant [tooling](https://github.com/JuliaQuantumControl/JuliaQuantumControl). Specifically, a [`install_org.jl`](https://github.com/JuliaQuantumControl/JuliaQuantumControl/blob/master/scripts/installorg.jl) script that gets included in all of the [projects’ test-tooling](https://github.com/JuliaQuantumControl/QuantumControl.jl/blob/24b58cdaa321c58977d65dbf3f3d197e46dc1e2a/devrepl.jl#L21-L26) and [all of their CI runs](https://github.com/JuliaQuantumControl/QuantumControl.jl/blob/24b58cdaa321c58977d65dbf3f3d197e46dc1e2a/.github/workflows/CI.yml#L38-L39). The script dev-installs all packages in the right order, either from a local checkout in a sibling folder or from the `master` branch on Github.

---

<div class="post-metadata">

**Author:** ![doit](https://avatars.discourse-cdn.com/v4/letter/d/9dc877/32.png) [@doit](https://discourse.julialang.org/u/doit)\
**Post date:** [February 15, 2022, 7:53am UTC](https://discourse.julialang.org/t/developing-concurrent-interdependent-packages-in-a-local-registry/76428/3 "2022-02-15T07:53:27Z")

</div>

Thanks for the quick answer, @goerz. Maybe I wasn’t explicit enough but I am already using LocalRegistry. The issue arises when I reason about what would happen when I change the master-branch.

You gave very good advise on that front. I had forgotten but I had already extracted another portion of my main code into its own package and I could quickly test what you talked about.  
Indeed, when I `dev` the modules using their path on disc (in the correct order, as you mentioned), the most up-to-date parts of the code are used and Julia does not rely on the version number in the Manifest. So that’s nice, I can at least partially test extracting the other part of my main package as well.

However, I also tested to see what happens when I actually `]test` the currently active (main) package. In that case, Julia automatically grabs the recent version as defined by the Manifest (in the `/test` folder) and relies solely on the local registry I created. So whatever changes I may make to a new package I extract from my main package, the automatic tests will fail.  
Is there a workaround such that even testing uses the modifed `dev`ed module?

~~I know that when I use a package not added to the test manifest that Julia throws a warning about that but runs anyway. Does the test then use the `dev`ed version? Or rather, the most up-to-date version (including `dev`) Julia knows globally about?~~  
~~Then at least for a new package\*, you could rapidly test code extraction using Julia’s automatic test system.~~  
**Edit:** Maybe I remembered old behaviour but Julia 1.7.0 errors on test if a package isn’t added properly. So that’s not an option.

\*I try to be careful with my wording here. “New package” means extracting a portion of a main package into its own package.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [February 15, 2022, 6:16pm UTC](https://discourse.julialang.org/t/developing-concurrent-interdependent-packages-in-a-local-registry/76428/4 "2022-02-15T18:16:37Z")

</div>

> I am already using LocalRegistry

Yes, sorry… I was speaking in general terms, for anyone comeing across this post. You’re good as far as using your own registry! 😉

> However, I also tested to see what happens when I actually `]test` the currently active (main) package.

What `Pkg.test()`, that is, `]test`, does is to create a new temporary project based on either the [legacy specification of the test environment in the main `Project.toml`](https://pkgdocs.julialang.org/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.0-and-1.1) or (preferred) based on the [`Project.toml` in the `test` subdirectory](https://pkgdocs.julialang.org/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.2-and-above). It then runs a new “sandboxed” Julia process with some special flags like `--check-bounds=yes` that simply does `include("test/runtests.jl")`. This happens here:

> <https://github.com/JuliaLang/Pkg.jl/blob/68ba0adc9b686db83dd65676cdd2c55313bd2520/src/Operations.jl#L1448>

This is why if you have a package `A` that depends on `B`, running `Pkt.test()` for `A` will use the version of `B` in the registry, instead of a local checkout of `B` as you want.

You basically have two choices:

- Put code in the `runtests.jl` file of package `A` that dev-installs `B`

- Don’t use `Pkg.test()`

I prefer the second solution. Put your test dependencies in `test/Project.toml`, and you can run

```julia
julia -p test
julia> Pkg.develop(path=path_to_package_B)
julia> Pkg.instantiate()
julia> include("test/runtests.jl")

```

If you want coverage information, bound checking, etc., you have to start `julia` with the appropriate flags.

If you want to be fancy (and need to do this across multiple projects) you can write your own version of `Pkg.test` that uses an existing environment (the `test/Project.toml` in `A`) instead of the sandboxed temporary environment it normally uses.

For my own usage, I’ve implemented

> <https://github.com/JuliaQuantumControl/QuantumControlBase.jl/blob/842f4a72a877c9e3ac9f67304ce2a55d430d6368/src/testutils.jl#L64>

which run `test/runtests.jl` in a subprocess and also gives me automatic coverage reports, via the very nice [LocalCoverage](https://github.com/JuliaCI/LocalCoverage.jl) package

---

<div class="post-metadata">

**Author:** ![doit](https://avatars.discourse-cdn.com/v4/letter/d/9dc877/32.png) [@doit](https://discourse.julialang.org/u/doit)\
**Post date:** [February 16, 2022, 7:31am UTC](https://discourse.julialang.org/t/developing-concurrent-interdependent-packages-in-a-local-registry/76428/5 "2022-02-16T07:31:43Z")

</div>

> [@goerz](#):
>
> Yes, sorry… I was speaking in general terms, for anyone comeing across this post. You’re good as far as using your own registry! 😉

No offense taken, I just wanted to be clear about that because using LocalRegistry is kind of part of the “problem”. I don’t mean to imply that a local registry is in and of itself a problem, not at all, it’s the only reasonable way of how to handle private packages that depend on each other (just as you mentioned). But then you force the system to use versioning which causes conflicts when you are required to develop two packages in tandem and need to test interoperability at an early stage. (At least at first glance.)

> [@goerz](#):
>
> What `Pkg.test()` , that is, `]test` , does is to create a new temporary project based on either the [legacy specification of the test environment in the main `Project.toml`](https://pkgdocs.julialang.org/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.0-and-1.1) or (preferred) based on the [`Project.toml` in the `test` subdirectory](https://pkgdocs.julialang.org/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.2-and-above). It then runs a new “sandboxed” Julia process with some special flags like `--check-bounds=yes` that simply does `include("test/runtests.jl")` .

Interestingly enough, after mulling over what you said, I tried simply activating a fresh environment and `dev`ed both packages. When I run a test on `A` in that environment, it works perfectly fine. The test will use the most up-to-date version of `B` without a hick. That way, I can test if I implement the interface between `A` and `B` correctly without having to commit to an updated version number.

That pretty much solves my issue (at least at a local level; I haven’t used any automated systems or pipelines and whatnot, so your other information will certainly come in handy then 😁). I’m unsure why it didn’t work globally but I like using environments to keep everything clean, so that’s a plus in any case. (Maybe I had package `A` active…)