How to install non-registered dependencies in Julia 0.7/1.0

tobias.knopp · August 23, 2018, 10:42pm

In Julia 0.6 it was possible to install non-registered dependencies while loading a package using this trick:

if !isdir(Pkg.dir("LinearSolver"))
  println("Installing LinearSolver...")
  Pkg.clone("https://github.com/tknopp/LinearSolver.jl.git")
end

In that case its my package LinearSolver that I use in a package called MPIReco (see here: https://github.com/MagneticParticleImaging/MPIReco.jl/blob/master/src/MPIReco.jl#L4)
The trick of course was to put LinearSolver not into REQUIRE.

Now with 0.7 I need to put LinearSolver into REQUIRE and in turn installation fails due to the missing dependency. Is there really not solution ( I am fine with a workaround ) to this problem. Travis runs on this package and in turn I need a solution the does not require manual tweaking.

I would appreciate any solution. My packages are upgraded to 0.7 on a branch but this issue prevents me merging things to master.

Thank,

Tobi

Keno · August 23, 2018, 11:30pm

Unregistered dependencies can be specified by git url in manifests, so if you start using Project/Manifests for your package that should work fine.

tobias.knopp · August 23, 2018, 11:33pm

Thanks, but its still not clear how to do that. Is there a tutorial how I can convert my REQUIRE based project to a Project/Manfest based one?

tobias.knopp · August 23, 2018, 11:37pm

Is this to activate my project and then add https://github.com/tknopp/LinearSolver.jl.git?

Keno · August 23, 2018, 11:40pm

yes

tobias.knopp · August 23, 2018, 11:41pm

I will try thanks!

tobias.knopp · August 23, 2018, 11:42pm

( I hope REQUIRE will go away soon. Currently it confuses the entire situation more )

tobias.knopp · August 25, 2018, 7:46am

I still do not get this working. I get

(v0.7) pkg> dev https://github.com/MagneticParticleImaging/MPIReco.jl.git
  Updating git-repo `https://github.com/MagneticParticleImaging/MPIReco.jl.git`
[ Info: Assigning UUID cffeea2d-44ef-50d9-9a8a-3f5d8306ed6d to MPIReco
[ Info: Path `/Users/knopp/.julia/dev/MPIReco` exists and looks like the correct package, using existing path instead of cloning
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package LinearSolver [12d7aa1e]:
 LinearSolver [12d7aa1e] log:
 ├─LinearSolver [12d7aa1e] has no known versions!
 └─restricted to versions * by MPIReco [cffeea2d] — no versions left
   └─MPIReco [cffeea2d] log:
     ├─possible versions are: 0.0.0 or uninstalled
     └─MPIReco [cffeea2d] is fixed to version 0.0.0

tobias.knopp · August 25, 2018, 8:05am

@kristoffer.carlsson: I have tried every information that I have found on Discourse but do not get it going.

I have these three packages:

All of them have no version, are unregistered, have a “julia-0.7” branch that is ported. I migrated REQUIRE to Manifest/Project by calling generate change version to 0.0.0 and than adding packages after activation.

In particular I did:

add https://github.com/MagneticParticleImaging/MPIFiles.jl.git#julia-0.7
add https://github.com/tknopp/LinearSolver.jl.git#julia-0.7

to mark the dependency of MPIReco in MPIFiles and LinearSolver. But I ran into the above error. I even tried master Pkg and that also did not resolve it.

tobias.knopp · August 25, 2018, 2:05pm

Do I need to make a release on unregistered projects? Is that the issue? In 0.6 one was just tracking the master branch without releases.

tobias.knopp · August 25, 2018, 7:08pm

I am totally lost here after trying out various things. Does anybody have a working example where package A depends on package B (both unregistered) and Travis runs successfully on packages A? Thats all I want.

Sukera · August 25, 2018, 7:54pm

When trying to add your MPIReco.jl package into a fresh environment, I already get this:

(tknopp) pkg> add https://github.com/MagneticParticleImaging/MPIReco.jl.git#julia-0.7
  Updating git-repo `https://github.com/MagneticParticleImaging/MPIReco.jl.git`
[ Info: Assigning UUID cffeea2d-44ef-50d9-9a8a-3f5d8306ed6d to MPIReco
 Resolving package versions...
ERROR: The following package names could not be resolved:
 * LinearAlgebra (not found in project, manifest or registry)
 * LinearSolver (not found in project, manifest or registry)
 * MPIFiles (not found in project, manifest or registry)
 * Random (not found in project, manifest or registry)
Please specify by known `name=uuid`.

Manually adding those dependencies let’s me add MPIReco. Maybe those dependencies are mucked up somehow in your .tomls?

Also, it may not be related, but I’ve noticed both LinearSolver and MPIFiles specify julia 0.6 in their travis configs.

tobias.knopp · August 25, 2018, 7:56pm

yes I just changed back to the REQUIRE version in order to test travis. So I am not sure how this can muck up tomls.

Sukera · August 25, 2018, 8:00pm

Ah, I see! I’ve just cloned MPIReco.jl manually, renamed Project and Manifest.toml back and did ]activate . and ]instantiate - I’m still building dependencies, but I’ll let you know if an install works that way.

EDIT: Well instantiating works as expected, st also shows the correct branches. Trying to dev the cloned git folder in a fresh environment gives the same Error you’ve seen before with the no versions left message.

tobias.knopp · August 25, 2018, 8:15pm

yep, thats why I am asking for help. I can develop locally but I am looking for a solution that I can deploy on travis.

Sukera · August 25, 2018, 8:19pm

github.com

JuliaLang/julia/blob/cdd4e84ac921ea7a40b3968440d35a37134bb11c/stdlib/Pkg/src/GraphType.jl#L583


      
          function init_log!(data::GraphData)
              np = data.np
              pkgs = data.pkgs
              pvers = data.pvers
              rlog = data.rlog
              for p0 = 1:np
                  p = pkgs[p0]
                  id = pkgID(p0, data)
                  versions = pvers[p0]
                  if isempty(versions)
                      msg = "$id has no known versions!" # This shouldn't happen?
                  else
                      msg = "possible versions are: $(VersionSpec(VersionRange.(versions))) or uninstalled"
                  end
                  first_entry = get!(rlog.pool, p) do; ResolveLogEntry(rlog.journal, p, "$id log:") end
          
          
        if p ≠ uuid_julia
                      push!(first_entry, (nothing, msg))
                      push!(rlog.init, (first_entry, ""), false)
                  end
              end

Well, looks like it does! This needs more investigating.

EDIT: I suspect travis won’t work without fixing the “regular” workflow for adding your package - there’s obviously something wrong with the dependency declarations (although they look fine and should all be found as they’re declared in the Manifest).

Ralph_Smith · August 25, 2018, 8:24pm

This is the sign of trouble. Pkg has dreamed up new UUIDs, ignoring the content of your repo Project files. Patching the environment Project and Manifest files with the correct UUIDs might help.

Sukera · August 25, 2018, 8:27pm

That’s unrelated - the packages are unregistered and that UUID is for the Top-level package they’re trying to add in the first place - it shouldn’t be referenced by any of its dependencies.

tobias.knopp · August 25, 2018, 8:31pm

I already use that UUID within the Project.toml of MPIReco.

cstjean · August 25, 2018, 8:41pm

tobias.knopp:

add https://github.com/MagneticParticleImaging/MPIFiles.jl.git#julia-0.7 
add https://github.com/tknopp/LinearSolver.jl.git#julia-0.7
to mark the dependency of MPIReco in MPIFiles and LinearSolver.

This is not what add X#julia-0.7 does; it says that the project (or package) depends on X, and that the environment (i.e. Manifest) should track the julia-0.7 branch. Since there’s only one environment active, that does not work as a general strategy (i.e. you could work in X’s environment, but then Y’s dependency on Z#julia-0.7 will be ignored). AFAIK, there’s no way to say that a package depends on some branch of another package. However, you can still have an environment in which all the branches you want are loaded: