I did pkg> develop for a couple of packages to work on making them compatible with 0.7. This included adding new dependencies (like say Nullables). However, doing a new pkg> add ... will wipe all the changes made to the developed packages deps. I suggest that pkg> add checks what packages are being developed and merges their Manifest.toml entries instead of overwriting them.
Could you elaborate a bit. It is not clear to me what is being overwritten and from where.
Sure. I’m working on developing a few of my packages, to upgrade them for v0.7. They have dependencies which have not yet been updated to support v0.7. Take for example DateParser which uses Nullables.
Remember that I don’t work on a project - I just want to get my packages to load into the Julia v0.7 REPL. So I start the REPL and do a using Genie and then debug the errors. Thus I’m talking about the global ~/.julia/environments/v0.7/Manifest.toml file.
And let’s say I get an error about Nullables in DateParser. I go into pkg> and do develop DateParser. I then edit the source file of DateParser and add using Nullables. But I also have to add Nullables to the deps list in Manifest.toml.
So the corresponding section of Manifest.toml now looks like this:
[[DateParser]]
deps = ["Dates", "Nullables", "TimeZones"]
path = "/Users/adrian/.julia/dev/DateParser"
uuid = "c513aa4d-7626-5656-a882-4604be010eff"
version = "0.1.1+"
Then I happily get back to work - and at some point I realize that I need to add a new package. Say OhMyREPL. So I go back to the REPL and run:
(v0.7) pkg> add OhMyREPL
Updating registry at `~/.julia/registries/Uncurated`
Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
Resolving package versions...
Updating `~/.julia/environments/v0.7/Project.toml`
[5fb14364] + OhMyREPL v0.2.11
Updating `~/.julia/environments/v0.7/Manifest.toml`
Now Nullables is gone from DateParsers list of deps and my code breaks:
[[DateParser]]
deps = ["Dates", "TimeZones"]
path = "/Users/adrian/.julia/dev/DateParser"
uuid = "c513aa4d-7626-5656-a882-4604be010eff"
version = "0.1.1+"
Note that I’ve also updated the REQUIRE file for DateParser, but that doesn’t seem to help either:
julia 0.7
TimeZones
Nullables
P.S. - thanks for all your hard work, I’m getting the hangs of Pkg3 and it’s pretty cool!
Hmmm, sorry… I figured it out. Actually updating the REQUIRE does help – but I updated the wrong REQUIRE. Instead of updating the one from ~/.julia/dev/DateParser, I edited the one in ~/.julia/packages/DateParser.
Sorry for the false alarm. It takes a bit of getting used to the ways of Pkg3 and it’s more of a discovery process.
So I guess my only comment would be to have some sort of upgrade workflow in the docs and mention that the REQUIRE file also needs to be updated, in the developed package, until fully migrated to Pkg3? I can probably write that, I think I now understand how it should be done.
Or rather, when doing pkg> develop why doesn’t Pkg3 update the cloned package? By adding the corresponding Manifest.toml file, to force an upgrade and get people to use the new system?
Update:
I probably still don’t get what’s the update package workflow. I just done pkg> develop on my own Genie as I’m working to migrate from HttpServer to HTTP. So I’ve done pkg> add HTTP in the Genie project (in ~/.julia/dev/Genie) and added using HTTP in the files but the dependency is still not satisfied:
ERROR: LoadError: LoadError: LoadError: ArgumentError: Module HTTP not found in current path.
Run `Pkg.add("HTTP")` to install the HTTP package.
require(::Module, ::Symbol) at ./loading.jl:868
include at ./boot.jl:314 [inlined]
include_relative(::Module, ::String) at ./loading.jl:1071
include(::Module, ::String) at ./sysimg.jl:29
include(::String) at /Users/adrian/.julia/dev/Genie/src/Genie.jl:4
top-level scope
include at ./boot.jl:314 [inlined]
include_relative(::Module, ::String) at ./loading.jl:1071
_require(::Base.PkgId) at ./loading.jl:997
require(::Base.PkgId) at ./loading.jl:878
require(::Module, ::Symbol) at ./loading.jl:873
eval at ./boot.jl:316 [inlined]
(::getfield(Distributed, Symbol("##164#166")){Module,Expr})() at ./task.jl:254
in expression starting at /Users/adrian/.julia/dev/Genie/src/Cookies.jl:6
in expression starting at /Users/adrian/.julia/dev/Genie/src/Genie.jl:39
Stacktrace:
[1] sync_end(::Array{Any,1}) at ./task.jl:221
[2] macro expansion at ./task.jl:240 [inlined]
[3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:206
[4] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:190
[5] include at ./boot.jl:314 [inlined]
[6] include_relative(::Module, ::String) at ./loading.jl:1071
[7] include(::Module, ::String) at ./sysimg.jl:29
[8] exec_options(::Base.JLOptions) at ./client.jl:251
[9] _start() at ./client.jl:427
in expression starting at /Users/adrian/Dropbox/Projects/HotelHound/hotel-hound/genie.jl:190
Update 2
I had to add HTTP as a dependency of Genie into the main Manifest.toml file to get it to work… Oh dear.
Don’t edit the Manifest by hand.
At this point, the current Manifest.toml has not had a chance to update. In other words, the Manifest.toml which list all the dependencies are out of sync with the actual dependencies. You can do dev Genie again to cause the manifest to update, or something like update --fixed. This is a common confusion and we are thinking about better ways to do this.
That worked, but now something else has happened.
1 - I’ve initially done pkg> develop Genie. No Project.toml nor Manifest.toml files were created for Genie dev.
2 - I’ve run pkg> init in Genie dev and got an empty Project.toml
3 - I’ve added a package (say Nullables). The requirements were not imported from REQUIRE into Project.toml and Manifest.toml – only Nullables was added to Genie dev Project.toml and Manifest.toml.
4 - now I’ve done pkg> develop Genie again to sync Genie’s dev deps with master Manifest and it wiped off all the dependencies that were initially imported from REQUIRE into the master Manifest.
So I think my previous comment is even more important here: when doing pkg> develop it should automatically create the Project.toml and Manifest.toml and set them up with the deps from REQUIRE. Then ideally remove the REQUIRE file so only one source of deps is left.
That’s not within the functionality of develop to do. Upgrading existing packages to have Project file is a wip (what is left is OS-specific deps and then everything is in place)
That seems correct, what did you expect to happen?
Yes because you created a project file in Genie which take precedence over the REQUIRE file.
I would’ve liked that:
a. Project.toml and Manifest.toml to be created. Since we’re migrating to the new Pkg, what’s the value in not creating them?
b. after creating them, to import the dependencies from REQUIRE into Project.toml - REQUIRE is the current authoritative source for the package’s dependencies. What is the benefit in not doing it?
None, which is why I said that there is WIP to create them for you. In order to do this properly though we need to support in the Project file what the current package manager support. What is left is OS-specific dependencies. After that is in, the next task is to have some upgrade function that does exactly what you want. Create a Project file based on the REQUIRE + test/REQUIRE files.
3 Likes
Awesome, thank you!
P.S. - a lot is lost when communicating in writing, so it’s not like I’m “demanding” things, just providing feedback. I know you guys have looked at all the angles but the users end up employing the APIs in completely unexpected manners. I respect and appreciate your work and I’m looking forward to Pkg3 maturing.
3 Likes