Pkg3 and standard libraries


#1

Encouraged by a commit message on the main repository I started using Pkg3 on my locally compiled version of master. I am working on revisions to the PDMats package for Julia v0.7 so I begin with

using LinearAlgebra

which is fine in the REPL but results in

julia> using PDMats
[ Info: Precompiling module PDMats
ERROR: LoadError: ArgumentError: Module LinearAlgebra not found in current path.
Run `Pkg.add("LinearAlgebra")` to install the LinearAlgebra package.

when precompiling the package. Also Pkg3 doesn’t seem to like dependencies on standard library names

pkg> up                   
[ Info: Updating registry at /home/bates/.julia/registries/Uncurated                                     
[ Info: Resolving package versions                  
ERROR: The following package names could not be resolved:                                                
 * LinearAlgebra (not found in project or manifest) 
Please specify by known `name=uuid`.                

Can someone help me return to the path of righteousness?


#2

Sorry, this is a bit off topic, but I thought Pkg3 was the default package manager for 0.7. Is that not the case, and if not, will it be before a release candidate?

At the risk of overstating things, it seems rather hugely important for the growth of the language as Pkg2 is really rather rough.


#3

Currently both Pkg and Pkg3 are in stdlib. I believe that Pkg3 is dormant until the user does something that results in certain directories being created under ~/.julia.


#4

So does that mean that Pkg is aliasing Pkg3 somehow? (I had assumed it would go by the name Pkg and there would be no Pkg3 module)

I frequent these forums and I see quite a lot of newcomers and experienced Julia users alike having some problem or another because their packages did not update correctly. In particular, a lot of people seem to still be on the old versions of DataFrames which I’m sure is rather frustrating for the DataFrames devs. This is why getting things on Pkg3 seems hugely important to me: stuff seems to routinely break that should work just fine because the package manager is wonky.

(sorry I feel like I’m derailing your thread, perhaps I should open another with this question?)


#5

I probably jumped the gun here in using Pkg3 in a situation that is not entirely straightforward. I seem to have developed adult-onset attention deficit disorder and am attracted to shiny new objects.


#6

Bad timing – you caught us between changing a bunch of names in the registry and merging the PR that changes the same names on Julia master:

We will announce on discourse when it’s all settled and it’s time to give Pkg3 a spin.


#7

Could I be greedy and ask for the announcement to have some updated documentation as well to make it easier to work with?


#8

Sure!


#9

Merged now, you can give it another try.


#10

Still no joy, unfortunately.

I compiled

 Version 0.7.0-DEV.4390 (2018-02-26 07:59 UTC)

and working with a fresh ~/.julia directory used the REPL switch ] to execute

status
add PDMats
checkout PDMats --branch=julia7

In the Julia REPL

using LinearAlgebra

is fine. If I add that in the PDMats/src/PDMats.jl file the precompilation fails as before

julia> using PDMats
[ Info: Precompiling module PDMats
ERROR: LoadError: ArgumentError: Module LinearAlgebra not found in current path.
Run `Pkg.add("LinearAlgebra")` to install the LinearAlgebra package.
Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:863
 [2] include at ./boot.jl:295 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1066
 [4] include(::Module, ::String) at ./sysimg.jl:29
 [5] top-level scope
 [6] eval at ./boot.jl:298 [inlined]
 [7] top-level scope at ./<missing>:3
in expression starting at /home/bates/.julia/dev/PDMats/src/PDMats.jl:6
ERROR: Failed to precompile PDMats to /home/bates/.julia/compiled/v0.7/PDMats/wuzE.ji.
Stacktrace:
 [1] error at ./error.jl:33 [inlined]
 [2] compilecache(::Base.PkgId) at ./loading.jl:1200
 [3] _require(::Base.PkgId) at ./loading.jl:1002
 [4] require(::Base.PkgId) at ./loading.jl:873
 [5] require(::Module, ::Symbol) at ./loading.jl:868

#11

What if you do e.g.

pkg> up --fixed

and then try using PDMats again.

The technical reason why this works is that you can only load stuff that is in your Manifest.toml and when you added using LinearAlgebra without resolving, the Manifest and the package are out of sync. Doing something like up should resync them. We have to figure out what the best way to do this will be.


#12

That worked, thanks.


#13

One more question, if I may. I am having trouble with

pkg> test Rmath
[ Info: Testing Rmath located at /home/bates/.julia/packages/Rmath/dy8D
ERROR: LoadError: LoadError: ArgumentError: Module Libdl not found in current path.
Run `Pkg.add("Libdl")` to install the Libdl package.     
...

I am able to test the SpecialFunctions package this way. I haven’t been able to decide what is different about SpecialFunctions compared to Rmath that gets it past the Libdl check.


#14

Keep 'em coming (you can also open issues at the Pkg3 repo). I’ll look at this soon.


#15
using Libdl

You can check it through the master branch under stdlib


Or through Compat


#16

If you wanna check out some work I did on PDMAts 0.7 version let me know. I converted all the source code and tests to 0.7 version and had all, but three files passing all tests.


#17

There is a problem here that when loading (precompiling) Rmath it doesn’t find Libdl when using Pkg3, which it should.


#18

It turns out that BinDeps injects a dependency on Libdl into packages that use it. So we need to hack this into the script that generates the Uncurated package registry from METADATA. Once that’s done, it will be possible to use packages that use BinDeps and actually rely on Libdl.


#19

Should work now after a pkg> up. Thanks for the report.


#20

All good now. Thanks again.