# Developing interdependent, local, unregistered packages

**URL:** <https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936>\
**Category:** General Usage\
**Tags:** question, package\
**Created:** [September 16, 2023, 9:04pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936 "2023-09-16T21:04:31Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [September 16, 2023, 9:04pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/1 "2023-09-16T21:04:31Z")

</div>

I’m trying to develop some local, interdependent, unregistered packages on Julia v1.9. As far as I can tell, it is still really hard to make it work. Am I doing something wrong? Is there a usable workflow on Julia v1.9? I guess I can try LocalRegistry.jl, but I was hoping I wouldn’t need that.

My dependency graph looks like this:

```julia
B ← A → C

```

where arrows point from dependency to dependent. So, B and C both depend on A but do not depend on each other. However, B is a test dependency for C. Is there a simple workflow using either `dev` or `add <github url>` that will get all the packages working, including the unit testing of package C, which depends on B?

I was just using `add <github url>`, which seemed to work fine until I tried to get testing for C to work, where the C unit tests depend on B. So, that seems to be the crux of the issue right now.

---

<div class="post-metadata">

**Author:** ![Ralph\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ralph_smith/32/10344_2.png) [@Ralph\_Smith](https://discourse.julialang.org/u/Ralph_Smith)\
**Post date:** [September 16, 2023, 9:29pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/2 "2023-09-16T21:29:19Z")

</div>

Maybe `Pkg.test("C"; allow_reresolve=false)` will work if you have `C/test/Manifest.toml` set up appropriately.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 16, 2023, 10:50pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/3 "2023-09-16T22:50:21Z")

</div>

Does `dev /path/to/B` in C’s environment work? (And then move the `[deps]` entry to `[extras]`.)

---

<div class="post-metadata">

**Author:** ![martin.d.maas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/martin.d.maas/32/50964_2.png) [@martin.d.maas](https://discourse.julialang.org/u/martin.d.maas)\
**Post date:** [September 17, 2023, 12:39am UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/4 "2023-09-17T00:39:27Z")

</div>

> [@tim.holy](#):
>
> Does `dev /path/to/B` in C’s environment work?

I second this. I generally use `dev` rather than `add`. If I’m not mistaken, I believe `dev` integrates better with Revise, so that way you can just change some of your code in B and get it immediately reflected on your current environment. When using `add` I think you need to commit your changes first, but I’m not quite sure right now. This is definitively not an issue if `B` doesn’t change often.

Remember you can also use relative paths… so `dev ../B` is also an option.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [September 17, 2023, 5:41am UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/5 "2023-09-17T05:41:53Z")

</div>

The only way I’ve found to make this work without jumping through crazy hoops is to register those packages in a [`LocalRegistry`](https://github.com/GunnarFarneback/LocalRegistry.jl).

As a general rule, any package that is a dependency (even a test dependency) for any other package should be registered in some registry. Luckily, working with `LocalRegistry` is really straightforward.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [September 17, 2023, 2:14pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/6 "2023-09-17T14:14:27Z")

</div>

> [@tim.holy](#):
>
> Does `dev /path/to/B` in C’s environment work? (And then move the `[deps]` entry to `[extras]`.)

It doesn’t seem to work. If I do that (dev B and move it to `extras` and `targets`) and then run `test` on the package C, I get the following error:

```julia
ERROR: expected package `B [1f83ef90]` to be registered

```

Furthermore, if I run `status`, I get this warning:

```plaintext
Warning The project dependencies or compat requirements
have changed since the manifest was last resolved. It is
recommended to `Pkg.resolve()` or consider `Pkg.update()`
if necessary.

```

Then if I run `resolve` or `update` the entry for the dev’d B package gets removed from the manifest file:

```julia
(C) pkg> update
    Updating registry at `~/.julia/registries/General.toml`
  No Changes to `~/projects/C/Project.toml`
    Updating `~/projects/C/Manifest.toml`
  [1f83ef90] - B v0.1.0 `~/projects/B`
  [56ddb016] - Logging
  [8dfed614] - Test

```

Once the dev’d B entry is removed from the manifest, there is no hope that Julia will be able to properly run the tests for package C.

I also tried using Project.toml/Manifest.toml files in the `test` directory, but I ran into the dreaded `ERROR: can not merge projects`. (This is not the first time I’ve encountered that error.) Luckily, I think the “can not merge projects” error [might be fixed](https://github.com/JuliaLang/Pkg.jl/pull/3276) on Julia 1.10.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 17, 2023, 2:20pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/7 "2023-09-17T14:20:51Z")

</div>

(EDITED) I typically use [TestEnv.jl](https://github.com/JuliaTesting/TestEnv.jl). Once you activate the test environment, you can

```julia
]dev ../C
;cd test
include("runtests.jl")

```

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [September 17, 2023, 2:31pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/8 "2023-09-17T14:31:10Z")

</div>

That example is different from my example in at least two ways:

- You didn’t move the A dependency to `extras` and `targets`.
- My test dependency graph for C uses two layers of dev dependencies: `A -> B -> C`.
  - I don’t know if it’s still an issue, but in the past Pkg struggled with more than one level of dev dependencies.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 17, 2023, 2:31pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/9 "2023-09-17T14:31:31Z")

</div>

You’re right, I edited my reply. It might make sense to report this as a Pkg bug; ideally the manifest should not lose track of the path for C if you’ve moved it to `[extras]`.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [September 17, 2023, 5:41pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/10 "2023-09-17T17:41:31Z")

</div>

Personally, what I’d really like to see is some implementation of this: [Store some location info for deps in Project.toml · Issue #492 · JuliaLang/Pkg.jl · GitHub](https://github.com/JuliaLang/Pkg.jl/issues/492) (or something better).

I have had multiple times now where I have had a cluster of interconnected, non-registered packages that I want to be able to work on, and I want my collaborators to be able to work on.

I really want to be able to just send them a github link and have them `dev` that package and have it automatically dev that whole local constellation of packages. The current system is a pretty bad user experience where my collaborator needs to go into a bunch of sub-directories and individually `dev` them in a specific order to satisfy the dependency-graph.

And before someone brings it up, no I specifically do **not** want to set up a local registry because I want me and my collaborators to be able to easily edit and update these packages without constantly registering new versions.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [September 17, 2023, 6:48pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/11 "2023-09-17T18:48:43Z")

</div>

Note that registering in a LocalRegistry is much more straightforward and lightweight compared to General: no waiting, no restrictions, just run `register()` and the new version immediately becomes available to everyone. This doesn’t seem a significant overhead, given the benefits of registering and versioning a package.

If both you and collaborators always need to dev/modify the same multiple packages at once, maybe they shouldn’t really be separate packages?

But yeah more generally I can see what you mean, and what can be simplified wrt current situation…

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 17, 2023, 7:35pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/12 "2023-09-17T19:35:07Z")

</div>

That’s even true if you have independent, registered versions. Monorepos should improve that collaborator experience?

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [September 17, 2023, 8:47pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/13 "2023-09-17T20:47:46Z")

</div>

> [@aplavin](#):
>
> Note that registering in a LocalRegistry is much more straightforward and lightweight compared to General: no waiting, no restrictions, just run `register()` and the new version immediately becomes available to everyone. This doesn’t seem a significant overhead, given the benefits of registering and versioning a package.

It means though you can’t really do a Revise based workflow right? Constantly hitting `register()` as I’m interactively working on a package is a step down in the user experience.

In my opinion this is just something Pkg currently handles poorly. It’s not a terribly big deal or anything, but it’s certainly annoying, and I’ve still never seen a satisfying solution other than just individually `dev`ing each package in the correct order.

> [@tim.holy](#):
>
> Monorepos should improve that collaborator experience?

Yes, but I don’t always want a mono-repo. TTFX has improved a lot, but some of the codebases I’ve shared with collaborators still have annoying long startup times, often for functionality that’s not often needed, so I usually prefer packages (especially because precompilation is now **slower** than before).

The way Pkg is currently designed though really does push me towards monorepos for this stuff which is not ideal. Again though, not the end of the world, just something that I hope improves.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 18, 2023, 12:51am UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/14 "2023-09-18T00:51:35Z")

</div>

Monorepos are orthogonal to packages, no? What you check out != how Pkg sees your code base. The key thing is that with a monorepo you can update all them them with a single commit.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [September 18, 2023, 5:25am UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/15 "2023-09-18T05:25:42Z")

</div>

> [@Mason](#):
>
> I want me and my collaborators to be able to easily edit and update these packages without constantly registering new versions.

I’d still register the packages in a LocalRegistry, but then on top of that write yourself an initialization script that dev-installs all the packages you want to develop together into a single environment. For some inspiration, see the [Development notes for the QuantumControl organization](https://github.com/JuliaQuantumControl#development) and the [`installorg.jl` script](https://github.com/JuliaQuantumControl/JuliaQuantumControl/blob/master/scripts/installorg.jl) that I’m using for the [dev environment](https://github.com/JuliaQuantumControl/JuliaQuantumControl/tree/master). I agree, though: Julia does not have good built-in tooling for this sort of thing.

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [September 20, 2023, 8:38am UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/16 "2023-09-20T08:38:07Z")

</div>

I have long struggled with local modules.

The easiest way that works for me is to modify `LOAD_PATH`:  
`push!(LOAD_PATH, path_to_local_modules)`, with all local modules in a subdirectory of `path_to_local_modules`.  
This doesn’t require that modules are “packages”; no requirement for `Project.toml`, `Manifest.toml`.

Although this is the simplest and just works, it is not recommended and misses out on benefits of more structured package environments.

I have tried `dev path_to_module`, but this is repetitive, not easily updated and requires one to navigate a sometimes tricky dependency order.

Somewhat naively, I was expecting or hoping to be able to specify _once_ where each local module is located, and then to be able to `add` that module to the environment of any other local module.

One way I expected this to work was to make use of stacked environments.  
`dev` each local module into say `LocalModules/Project.toml`, which would act as a kind of local registry.  
With this “local registry” environment loaded into the current environment stack, one could just `add LocalModuleX` to the environment of any other `LocalModuleY`.

I’m not sure why this couldn’t work?

---

<div class="post-metadata">

**Author:** ![Olivier\_Merchiers](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/olivier_merchiers/32/4073_2.png) [@Olivier\_Merchiers](https://discourse.julialang.org/u/Olivier_Merchiers)\
**Post date:** [September 20, 2023, 12:03pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/17 "2023-09-20T12:03:03Z")

</div>

To me, this problem is one of my biggest pain point with Julia.  
I wish this problem appeared in the list of biggest gripes in the annual julia survey but it’s not so I cannot tick it.

It is very hard to do properly, using a combination of packages, dev command and local registry.

For an experienced julia developper this can be challenging, for a beginner, it’s virtually impossible to find out how to do this correctly.

Especially since the easiest solution using `push!(LOAD_PATH, path_to_local_modules)` is systematically discouraged.

I think this problem could be alleviated if [issue 4600](https://github.com/JuliaLang/julia/issues/4600#issue-21336090) got resolved, but not sure. I think this issue is more related with the behaviour of include.

I always wondered if it would be possible to implement `dev path/to/local_module` where `local_module` is just a module without being a package. This to me would be a good replacement for the `push!(LOAD_PATH, path_to_local_modules)` solution.

---

<div class="post-metadata">

**Author:** ![ianfiske](https://avatars.discourse-cdn.com/v4/letter/i/58f4c7/32.png) [@ianfiske](https://discourse.julialang.org/u/ianfiske)\
**Post date:** [September 20, 2023, 2:20pm UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/18 "2023-09-20T14:20:53Z")

</div>

I agree that this is difficult to get working without a LocalRegistry. I have had luck with the following approach to get local A and B availalble in C’s tests.

- Use a `test/Project.toml` and `test/Manifest.toml` (test-specific environment) instead of the `targets` way of adding test-specific dependencies.
- `dev /path/to/A` and `dev /path/to/B` in C’s test environment
- Use [TestEnv.jl](https://github.com/JuliaTesting/TestEnv.jl) to run your C tests.

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [June 20, 2024, 10:09am UTC](https://discourse.julialang.org/t/developing-interdependent-local-unregistered-packages/103936/19 "2024-06-20T10:09:19Z")

</div>

Oh, man. I’m currently struggling with the same: trying to come up with a monorepo layout that’s convenient to work with, can support multiple developers, and doesn’t require jumping through a lot of hoops.

Please note that there is a concept of [`package directories environment`](https://docs.julialang.org/en/v1/manual/code-loading/#Package-directories), whereby adding a directory to the LOAD\_PATH, you can load packages directly from subdirectories of the main package dir. But it doesn’t really work as advertised, mainly, IMHO, because the package manager doesn’t seem to be aware of this concept at all. In particular:

- it can’t add packages from this environment to your active project;
- it can’t resolve external dependencies (from the `General` registry) in the packages from this environment;
- once you add a package `A` to the dependency of package `B` within the environment, it can’t update its `Project.toml` interactively anymore, because it will always fail to identify `A`.

I wish such package directories envrionment were handled more similarly to the stdlib (which is also just a monorepo of interdependant packages, in a way).