"can not merge projects" error when `dev`ing sister package from test project

Tooling Package Management
1

I have run into an issue in trying to apply a project structure I have seen in some successful projects. Similar to StaticArrays.jl and StaticArraysCore.jl, I have MyProject.jl and MyProjectCore.jl, each with their own Git repository and Project.toml. So, picture the following directory structure on the local workstation (irrelevant files omitted):

git_repos
├── MyProjectCore.jl
│   ├── Project.toml
│   └── README.md
└── MyProject.jl
    ├── Project.toml
    ├── README.md
    └── test
        └── runtests.jl

(It is fairly easy to mock this up using PkgTemplates.jl but hard to show the interactive input in this forum post.)

MyProject.jl has MyProjectCore.jl as a dev dependency (julia --project in MyProject.jl/, then in REPL ]dev ../MyProjectCore.jl.

This works just fine, but as I continued to build out MyProject.jl, I realized I wanted to add (for example) DataFrames.jl as a test dependency. That is, DataFrames.jl is needed to run the tests for MyProject.jl, but is not needed for MyProject.jl itself, nor for MyProjectCore.jl.

So, I created MyProject.jl/test/Project.toml by running julia --project in MyProject.jl/, then in the REPL ]activate test followed by ]add DataFrames. No problems so far.

But now, I realized that I want to access some of the methods from MyProjectCore.jl directly in the MyProject.jl tests. Before, when I didn’t have a MyProject.jl/test/Project.toml, this was easy, because I was already deving MyProjectCore.jl. Now, I need to add MyProjectCore.jl as a dev dependency of the test projects, so again, from MyProject.jl/ I run ]activate test and dev ../MyProjectCore.jl.

This is where the trouble begins: If I now reactivate the MyProject.jl project (]activate .) and attempt to ]test it, I get the infamous

ERROR: can not merge projects

error.

Is there a way to make this project and dependency structure work?

2

Is this worth filing a bug for, or is my workflow misguided? If the former, I can put together a sample github repo for reproducibility.

3

This is a well-known error, which often pops up when you have non-standard test workflows, see "cannot merge projects" error · Issue #1585 · JuliaLang/Pkg.jl · GitHub
Usually one solution is to avoid test/Project.toml and put everything in the main Project.toml file using the [extras] and [targets] fields

4

Which version are you getting the error on? In general, I don’t think you need to ]dev the package under test in your tests. I use a test/Project.toml-based workflow and I never ]dev the main package in there.

5

Thanks! I saw that issue and initially thought it didn’t apply because of this explanation:

Yeah that is a pretty bad error message. It happens when the active manifest and the test manifest both have fixed nodes that disagree (for example the active manifest and the test manifest both are tracking the same UUID by path but are tracking it a different paths). Pkg has to preserve fixed nodes but it doesn’t know which one to keep.

I get the error even when I ]dev with absolute paths, so this seems not to be the full story. But the suggested workaround of using MyProject.MyProjectCore in the tests of the former worked for me, so I appreciate the link.

@Sukera v1.10.4. Note that I am not trying to ]dev the package being tested (MyProject.jl) but rather a sibling project (MyProjectCore.jl) that MyProject.jl depends on. To use a well-known example, this would be like using methods from StaticArraysCore.jl in the tests for StaticArrays.jl on a workstation where you are developing both packages.

6

@maxkapur - can you add a little detail on what you mean by using MyProject.MyProjectCore in tests?

I’m having the same issue with “sibling projects”; I followed the link for issue #1585 but can’t track what you refer to. Thanks!

7

If memory serves, the precise comment in the GitHub issue that got me to the solution was this one:

https://github.com/JuliaLang/Pkg.jl/issues/1585#issuecomment-1092256497

Unfortunately, I don’t remember the exact details or have the code in front of me (this was a work project that we ended up having to port to Python/NumPy).

8

Thank you for the reply!