Your situation is that the project has a dependency on PackageA, and PackageA has a [compat] section that you would like to look differently, right? In that case you simply devPackageA in the main project. At that point you can have a local editable copy of PackageA somewhere on your filesystem and your main project now points to that version of PackageA and you can make any changes you want there, including changes to the Project.toml of PackageA. Once you’ve made your changes, you can run pkg> update and all the other deps in the main project will update, now taking the new [compat] section in PackageA into account. And none of this will have any impact on anything else on your system.
How do I make sure I include my locally modified packageX (using dev --local packageX), which isn’t currently submitted to git, in some other packageY (which has packageX as its deps in project.toml file)?
I activated the environment for packageY. Then dev --local packageX. However, packageY still does not know about my locally modified packageX and keeps pointing to what I have in the git repo. I did change the Project.toml file of packageX and changed its version number.
packageX and packageY are both deps of packageZ, but I don’t care about packageZ yet. I created the dev versions of packageX and packageY by first activating environment for packageZ and then dev --local packageX, packageY. Yes, packageZ is aware of my local changes in packageX and packageY, which is what the above answer addresses, but I am interested in getting the local changes (including its src and toml files) in packageX reflected when I use packageY (since packageX is deps of packageY).
So far I’ve tried activating packageY. Then dev --local packageX (even though I had already dev --local packageX, Y while packageZ env was activated since they both are packageZ deps).
Do I really need to first free packageX, packageY which were dev’d while packageZ was activated?
How do I get the local changes in packageX reflected when I run packageY?
I’m not sure I entirely understand where it goes wrong for you and what the problem is.
Have you checked that the Manifest.toml of package Y contains the expected local path for package X?
It sounds like this should work, here is an example of how I create a new package MyTest, dev an existing package and update the version number of it which is then reflected in MyTest after running up.
Now we have StaticArrays in the folder of MyTest/dev/StaticArrays and we can make changes there. I update the version in Project.toml from 1.3.0 to 1.3.1 and then run up to see the changes reflected in the environment.
Thanks for the detailed explanation. This is what I was expecting with my case as well. There is one difference however. That “StaticArray” package (I’ve a diff package of course) in my case is part of a custom registry and it has compat.toml file which doesn’t cover the newly modified version number (of StaticArray package). Does it matter what is defined in the (custom) registry either for MyTest package or for StaticArray package when we are locally developing/modifying these packages?
The error I encounter is the following:
‘ERROR: empty intersection between StaticArray@0.6.0 and project compatibility 0.6.1-0.6’
Does this mean we need to edit custom registry as well? Is it recommended to edit custom registry when we are locally editing packages?
EDIT: Another question that confuses me is that when I add a packageA to my current project’s Project.toml file, I cannot seem to use “'” using pacakgeADep"'" in a julia script of my current project. Note that packageADep is the dependency package of packageA and packageA’s Project.toml files already includes this package as its dependency under [deps] section. I am a bit confused why I need to explicitly add pacakgeADep in my current project’s Project.toml file, since this is already a dependency of the packageA, which I have already added.
Okay, I should have used the packages X,Y,Z to keep it similar, but I just used something random when I tried it in the REPL and then I was too lazy to update it …
But here you are saying that X is in the custom registry, and it has a compat section in the Project.toml that is not compatible with its own version? I think I’m missing something here.
Which is the active project when you get this?
No, it should not. If you have the files locally and use dev to add them to the project they should only care about the local versions (in my understanding, and in all tests I have done).
Let’s just look at X and Y now, both are in the registry, but you want to do local changes? I would clone project to somewhere/Y and then do
] # Repl mode
activate somewhere/Y
dev --local X
# Update X in somewhere/Y/dev/X/...
up # Should now make sure version updates are reflected
This seems reasonable to me, since the active project is not package A, so the active project is not defined to necessary have any of package A’s dependencies. This would be problematic if for example package A realises they don’t need a dependency and remove it, but you used it through package A where it is now no longer available. So your package should define all dependencies itself to allow for a robust dependency tree.
Let’s just look at X and Y now, both are in the registry, but you want to do local changes? I would clone project to `somewhere/Y` and then do
] # Repl mode
activate somewhere/Y
dev --local X
# Update X in somewhere/Y/dev/X/...
up # Should now make sure version updates are reflected
Yes, this is what I want and yes, both X and Y packages are defined in custom registry with their compat.toml files, which, based on your response above, is not going to be relevant since we are developing packages locally.
So right after dev --local X, I go to X’s Project.toml file and change the version number. Then I go to Y’s Project.toml, and under [compat] section, I change the version number of X (so that Y uses newer local version X). During this process, Y’s environment is active, meaning, I have (Y) pkg>.
I then run (Y) pkg> up. Right here I get the error I reported above.
I would completely avoid the entire “activating packages” thing, I think for your situation there is no reason at all to ever activate either package X or Y.
Instead, the general workflow would be that you clone whichever package you want to change the compat section in somewhere locally, make that change in that folder, and then activate the main project that you started with (which is not one of the packages), then do pkg> dev PATH_TO_PACKAGE_X and pkg> dev PATH_TO_PACKAGE_Y (or whichever packages you have locally now) in the project that you are working with and then do a pkg> up. That will resolve all dependencies in the project you care about, i.e. this main project will not use the deved versions of package X and Y, and use the compat section of those deved packages to resolve all the other dependencies.
I’ve seen a ton of confusion with folks that start to work with packages about this aspect that any package is also a project and can be activated. It is one of the aspects that I wish were different in the package manager: I think a much cleaner design would be if packages had a Package.toml in their root (instead of using the same filename for package definitions and project definitions), and one simply couldn’t activate a package. Conceptually there would be a lot less overloading of terms/steps and the world would be much cleaner: there are packages, there are projects, and they are different things and there is no mix and match between these concepts.
@davidanthoff Thanks for this suggestion. I will try this and report back here. That said, I would be very confused if it solves this problem since I don’t see the diff between a directory created after (Y) pkg> dev --local X and git clone of X. Y and X are both packages of higher level project Z, which has both X and Y as deps. I activated project Z, then (Z) pkg> dev --local Y. Then activated this dev version of Y, and then (Y) pkg> dev --local X. This is how I ended up with local dev folders for X and Y packages. From here on I edited Project.toml of X and Y, and also edited the [compat] section in Project.toml of Y.
The exact line I heard in the video you shared above from JuliaCon2019: “Every package is a project, but not all projects are packages”.
Things can only be deved in a project, not in a package. So when you do (Y) pkg> dev --local X, then you dev X in the Yproject, but that will not cause X to be deved in the Z project, you have to dev X explicitly in the Z project for Z to pickup your locally cloned version of X. The fact that you deved X while you had the Yproject active has zero effect on which version of X is loaded when you have Z active. In particular, if you have Y deved in Z, and the the Y project has X deved, then that does NOT mean that X is deved in Z, because the Y project is completely ignored when the Z project is active.
I think all of this is super confusing, and I’ve seen very, very many folks get thrown off by this. That is the primary reason I think this dual project/package story is not a good design and wish there had been a clearer difference between packages and projects.
@davidanthoff , sorry for the confusion, but I am not sure if I agree with you here (I definitely appreciate your suggestions). Why is “Z” relevant in the discussion here? I simply gave you background info on Z to tell you how I got local copies of X and Y. Once I have the local copies of Y and X (where X is the deps of Y), why should it matter how I got the local copies of Y and X? I should be able to activate project Y and edit its toml files or src code, and similarly, again, (Y) pkg> dev --local X (notice Y is activated here) and modify X’s toml files and src code.
I would think that the “dev” command is simply git clone and some additional commands to update Y’s tomls to point to deved X, under the hood.
Please note that my original question was with respect to packages X and Y (these are truly packages in Julia’s sense). I changed the version of X, after I deved X while Y was active. Then I changed the [compat] of Y and tried running 'up" command, which gave me the error above. Why is Z relevant in any of this? I am confused.
Ah, wait, I was still in the mindset of you having gotten some “parent” project that contained all of these packages and how you would make sure to use all the deved packages from that project.
The ERROR: empty intersection between StaticArray@0.6.0 and project compatibility 0.6.1-0.6 error (that is the one you are referring to, right?) to me looks like you have pinned StaticArray in whatever project you have activated, and then there is a compat in one of the packages that is in this project that requires at least 0.6.1 for StaticArray?
@davidanthoff I have Y and X packages (how I got these packages locally is not useful info I guess, but I mentioned above how I got the local, editable versions of these packages). I activate Y and dev local X. If I run ‘status’, I can see dev’d version of X in the list, so things are fine up to this point. I then change the version number of X in its Project.toml file. Now, this version number is not covered in Y’s Project.toml file (under [compat] section, note that X is deps of Y), so I edit Y’s Project.toml file as well and update the version number for dependency X under the [compat] section. Finally, I run ‘up’ command (notice I didn’t run ‘up’ yet since I had to modify Y’s toml file as well so that I use the updated version number for X). I encounter error right here (error message in previous messages). As you can see, X is not pinned to older version in Y’s toml file. I do edit Y’s Project.toml file to use the updated version of X.
X and Y are both custom packages are are part of custom registry. Please ignore StaticArray package and let’s just use X and Y.
Hm, that is a bit strange… Can you try to run (Y) pkg> resolve and then (Y) pkg> up? I think I’ve run into scenarios in the past where I had to do that. I don’t fully understand why that would be necessary, but it did help for me in some cases.
And you are sure that you have incremented the version in X to at least 0.6.1, right? And it is not at 0.6.0 still? Are you sure that your Y project is pointing to the same local folder where you are updating the Project.toml in X? I would double check that (Y) pkg> status shows you the same local path for X where you are editing the Project.toml for `X. Other than that, I’m a bit lost. I do think what you are doing should work…
@davidanthoff, yes, I have incremented the X version to 0.6.1 (setting it back to 0.6.0, everything works fine, but I want to increment it to 0.6.1). After updating X’s version in its Project.toml file, I go back to Y’s Project.toml file and update the version under [compat] section, then run the (Y) pkg> up, but keep getting that error.
@aplavin, I deleted Manifest.toml files of both Y and X projects, restarted julia, etc. to no avail. When I run the (Y) pkg> up command, my custom registry also gets updated :
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General`
Updating registry at `~/.julia/registries/Custom`
Updating git-repo `git@gitlab.com:custom-julia-registry.g
ERROR: empty intersection between X@0.6.0 and project compatibility 0.6.1-0.6
EDIT: I confirm that if X and Y are random projects, not defined in custom registry, then everything works fine. However, if X and Y are custom projects and are defined in custom registry, then it just does not seem to work, despite the fact the X is dev’d when Y was activated and all
@jlapeyre Not sure if that is necessary since both X and Y are local packages. Package Y was cloned and then package X was devd while Y was activated: (Y) pkg> dev —local X.
…