How to upgrade my non-registered package to v0.7

floswald · September 13, 2018, 9:36pm

hi all

really sorry for the 528-th question about Pkg3. It’s not working for me.
I want to upgrade this GitHub - floswald/SMM.jl: Simulated Method of Moments for Julia to v0.7. Here is what I have done:

first try

start julia 0.7
] add https://github.com/floswald/MomentOpt.jl
] dev MomentOpt

(v0.7) pkg> dev MomentOpt
ERROR: The following package names could not be resolved:
 * MomentOpt (bc769cb7-f249-5bba-802a-79f18cb247ec in manifest but not in project)
Please specify by known `name=uuid`.

] rm MomentOpt

second try

] dev https://github.com/floswald/MomentOpt.jl
(v0.7) pkg> activate /Users/florian.oswald/.julia/dev/MomentOpt
(MomentOpt) pkg> test MomentOpt

ERROR: The following package names could not be resolved:
 * MomentOpt (not found in project or manifest)
Please specify by known `name=uuid`.

Do I need to add the package again to it’s own project? I thought I could create a manifest and project file for MomentOpt now? I can’t even load the code here?

(MomentOpt) pkg> rm MomentOpt

┌ Warning: `MomentOpt` not in project, ignoring

Ah. Ok.

shell> rm -rf ~/.julia/dev/MomentOpt/

third try

cd ~/.julia/v0.6/MomentOpt
git branch v0.7 && git checkout v0.7
start julia 0.7
(v0.7) pkg> dev .
(v0.7) pkg> test MomentOpt

That worked!

Question

That’s probably not the ideal workflow. what is?
now suppose I am finished with the upgrade. I push the branch to github. merge. should i then just add the package from github?

thanks!

tkoolen · September 14, 2018, 12:27am

Your second try is close. This does work:

(v1.0) pkg> dev https://github.com/floswald/MomentOpt.jl
(v1.0) pkg> test MomentOpt

Using ] status after ] activate /Users/florian.oswald/.julia/dev/MomentOpt (or simply ] activate MomentOpt) shows something like

(MomentOpt) pkg> status
    Status `~/.julia/dev/MomentOpt/Project.toml`

That is, the package manager now expects the Project.toml and Manifest.toml files in the MomentOpt directory to contain the information regarding what the MomentOpt package is, but MomentOpt does not contain a Project.toml or Manifest.toml file. Rather, that information is stored in the default environment, ~/.julia/environments/v1.0/Project.toml in my case. So by activating MomentOpt the package manager ironically loses the information regarding what package MomentOpt refers to.

Actually, you’d think that you should still be able to test MomentOpt because the default environment is still on the LOAD_PATH; in fact, using MomentOpt works. However, test MomentOpt doesn’t for some reason and that does seem inconsistent. There might be an issue in Pkg.jl for this already, but if not it’s probably a good idea to open one.

I personally use non-default environments that are stored separately from the packages, i.e. I have a directory ~/code/julia-projects that contains a subdirectory for each environment. To make a new environment, I’d do:

cd ~/code/julia-projects
mkdir newproject && cd newproject
julia --project

Then in Julia:

(v1.0) pkg> activate . # this is unfortunately currently needed despite --project (only) if there is no Project.toml file yet

(newproject) pkg> dev https://github.com/floswald/MomentOpt.jl

(newproject) pkg> test MomentOpt

fredrikekre · September 14, 2018, 7:13am

I am pretty sure test only looks in the current project and not the load path, which seems fine. In this case the “problem” is not Pkg, but rather that MomentOpt does not have a Project.toml file.

You can actually instead start julia with

julia --project=.

floswald · September 14, 2018, 8:10am

thanks both. I just tried again on my other machine. I understand much better already, but there is still somethings that don’t make sense:

  MomentOpt git:(v0.7) julia7 --project=.
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0 (2018-08-08 06:46 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-apple-darwin14.5.0

(MomentOpt) pkg> test
ERROR: trying to test unnamed project

(MomentOpt) pkg> test MomentOpt
ERROR: The following package names could not be resolved:
 * MomentOpt (not found in project or manifest)
Please specify by known `name=uuid`.

(MomentOpt) pkg> activate .

(MomentOpt) pkg> st
    Status `Project.toml`

(MomentOpt) pkg> dev .
[ Info: Assigning UUID bc769cb7-f249-5bba-802a-79f18cb247ec to MomentOpt
 Resolving package versions...
 Installed MiniLogging ──── v0.0.2
 Installed DataFramesMeta ─ v0.4.0
 Installed Missings ─────── v0.2.10
 Installed RecipesBase ──── v0.5.0
 Installed JLD2 ─────────── v0.1.2
 Installed Roots ────────── v0.7.2
 Installed Plots ────────── v0.19.3
  Updating `Project.toml`
  [bc769cb7] + MomentOpt v0.0.0 [`.`]
  Updating `Manifest.toml`
  many packages...

so far so good. now what about that:

(MomentOpt) pkg> test MomentOpt
   Testing MomentOpt
ERROR: expected a `name` entry in project file at /Users/74097/.julia/v0.6/MomentOpt/Project.toml

indeed,

shell> cat Project.toml
[deps]
MomentOpt = "bc769cb7-f249-5bba-802a-79f18cb247ec"

why does dev . not create a valid project.toml for me? I understand this is missing a [name] tag.

fredrikekre · September 14, 2018, 8:24am

dev adds a package from a folder, you have now basically added MomentOpt to itself.

The problem here is that you have activated an environment (MomentOpt) but that environment is not recognized as a package, since it does not have a name in its Project.toml file. Try to remove the Project.toml + Manifest.toml file you have in your MomentOpt folder, then you add a new (package-valid) Project.toml file, with e.g.

name = "MomentOpt"
uuid = "bc769cb7-f249-5bba-802a-79f18cb247ec"

and then start over.

floswald · September 14, 2018, 8:30am

thanks. not sure I did the right thing:

➜  MomentOpt git:(v0.7) ✗ ls
LICENSE.TXT   Manifest.toml README.md     appveyor.yml  histogram.png main.jl       slices-m.png  slices-v.png  test
Makefile      Project.toml  REQUIRE       docs          lines.png     proposals.gif slices-m2.png src
➜  MomentOpt git:(v0.7) ✗ rm *.toml
➜  MomentOpt git:(v0.7) vim Project.toml
➜  MomentOpt git:(v0.7) ✗ julia7 --project=.
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0 (2018-08-08 06:46 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-apple-darwin14.5.0

(MomentOpt) pkg> st
Project MomentOpt v0.0.0
    Status `Project.toml`

(MomentOpt) pkg> activate .

(MomentOpt) pkg> st
Project MomentOpt v0.0.0
    Status `Project.toml`

(MomentOpt) pkg> dev .
┌ Warning: project file for MomentOpt at . is missing a `version` entry
└ @ Pkg.Types /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v0.7/Pkg/src/Types.jl:700
ERROR: Cannot develop package with the same name or uuid as the project

(MomentOpt) pkg>

floswald · September 14, 2018, 8:36am

added a version number:

➜  MomentOpt git:(v0.7) ✗ julia7 --project=.
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0 (2018-08-08 06:46 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-apple-darwin14.5.0

(MomentOpt) pkg> activate

(MomentOpt) pkg> st
Project MomentOpt v0.1.0
    Status `Project.toml`

(MomentOpt) pkg> dev .
ERROR: Cannot develop package with the same name or uuid as the project

fredrikekre · September 14, 2018, 8:36am

That seems to be correct, note that you now are in a named project since status shows Project MomentOpt, compared to before, when you were in a regular unnamed environment.

Don’t do this, this tries to add MomentOpt to MomentOpt, you don’t need to add the package to itself.

floswald · September 14, 2018, 8:45am

great! works! thanks for your patience.

final question. I can’t get rid of these warnings about precompilation. are these indications about deprecated calls, i.e. do i need to do something here?

(MomentOpt) pkg> test MomentOpt
   Testing MomentOpt
 Resolving package versions...
WARNING: eval from module Plots to MomentOpt:    
Expr(:call, Expr(:., :Sys, :(:islinux)))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Plots to MomentOpt:    
Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1368 =#, Expr(:call, Expr(:., :Sys, :(:isapple))))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Plots to MomentOpt:    
Expr(:function, Expr(:call, :_show, Expr(:::, :io, :IO), Expr(:::, Expr(:curly, :MIME, Expr(:call, :Symbol, "application/postscript"))), Expr(:::, :plt, Expr(:curly, :Plot, :GRBackend))), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1379 =#, Expr(:call, Expr(:., :GR, :(:emergencyclosegks))), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1380 =#, :filepath = Expr(:call, :*, Expr(:call, :tempname), ".", "ps"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1381 =#, :env = Expr(:call, :get, :ENV, "GKSwstype", "0"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1382 =#, Expr(:ref, :ENV, "GKSwstype") = "ps", #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1383 =#, Expr(:ref, :ENV, "GKS_FILEPATH") = :filepath, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1384 =#, Expr(:call, :gr_display, :plt, "ps"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1385 =#, Expr(:call, Expr(:., :GR, :(:emergencyclosegks))), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1386 =#, Expr(:call, :write, :io, Expr(:call, :read, :filepath, :String)), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1387 =#, Expr(:call, :rm, :filepath), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1388 =#, Expr(:if, Expr(:call, :!=, :env, "0"), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1389 =#, Expr(:ref, :ENV, "GKSwstype") = :env), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1391 =#, Expr(:call, :pop!, :ENV, "GKSwstype")))))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Plots to MomentOpt:    
Expr(:function, Expr(:call, :_show, Expr(:::, :io, :IO), Expr(:::, Expr(:curly, :MIME, Expr(:call, :Symbol, "application/pdf"))), Expr(:::, :plt, Expr(:curly, :Plot, :GRBackend))), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1379 =#, Expr(:call, Expr(:., :GR, :(:emergencyclosegks))), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1380 =#, :filepath = Expr(:call, :*, Expr(:call, :tempname), ".", "pdf"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1381 =#, :env = Expr(:call, :get, :ENV, "GKSwstype", "0"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1382 =#, Expr(:ref, :ENV, "GKSwstype") = "pdf", #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1383 =#, Expr(:ref, :ENV, "GKS_FILEPATH") = :filepath, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1384 =#, Expr(:call, :gr_display, :plt, "pdf"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1385 =#, Expr(:call, Expr(:., :GR, :(:emergencyclosegks))), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1386 =#, Expr(:call, :write, :io, Expr(:call, :read, :filepath, :String)), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1387 =#, Expr(:call, :rm, :filepath), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1388 =#, Expr(:if, Expr(:call, :!=, :env, "0"), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1389 =#, Expr(:ref, :ENV, "GKSwstype") = :env), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1391 =#, Expr(:call, :pop!, :ENV, "GKSwstype")))))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Plots to MomentOpt:    
Expr(:function, Expr(:call, :_show, Expr(:::, :io, :IO), Expr(:::, Expr(:curly, :MIME, Expr(:call, :Symbol, "image/png"))), Expr(:::, :plt, Expr(:curly, :Plot, :GRBackend))), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1379 =#, Expr(:call, Expr(:., :GR, :(:emergencyclosegks))), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1380 =#, :filepath = Expr(:call, :*, Expr(:call, :tempname), ".", "png"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1381 =#, :env = Expr(:call, :get, :ENV, "GKSwstype", "0"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1382 =#, Expr(:ref, :ENV, "GKSwstype") = "png", #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1383 =#, Expr(:ref, :ENV, "GKS_FILEPATH") = :filepath, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1384 =#, Expr(:call, :gr_display, :plt, "png"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1385 =#, Expr(:call, Expr(:., :GR, :(:emergencyclosegks))), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1386 =#, Expr(:call, :write, :io, Expr(:call, :read, :filepath, :String)), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1387 =#, Expr(:call, :rm, :filepath), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1388 =#, Expr(:if, Expr(:call, :!=, :env, "0"), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1389 =#, Expr(:ref, :ENV, "GKSwstype") = :env), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1391 =#, Expr(:call, :pop!, :ENV, "GKSwstype")))))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Plots to MomentOpt:    
Expr(:function, Expr(:call, :_show, Expr(:::, :io, :IO), Expr(:::, Expr(:curly, :MIME, Expr(:call, :Symbol, "image/svg+xml"))), Expr(:::, :plt, Expr(:curly, :Plot, :GRBackend))), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1379 =#, Expr(:call, Expr(:., :GR, :(:emergencyclosegks))), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1380 =#, :filepath = Expr(:call, :*, Expr(:call, :tempname), ".", "svg"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1381 =#, :env = Expr(:call, :get, :ENV, "GKSwstype", "0"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1382 =#, Expr(:ref, :ENV, "GKSwstype") = "svg", #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1383 =#, Expr(:ref, :ENV, "GKS_FILEPATH") = :filepath, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1384 =#, Expr(:call, :gr_display, :plt, "svg"), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1385 =#, Expr(:call, Expr(:., :GR, :(:emergencyclosegks))), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1386 =#, Expr(:call, :write, :io, Expr(:call, :read, :filepath, :String)), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1387 =#, Expr(:call, :rm, :filepath), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1388 =#, Expr(:if, Expr(:call, :!=, :env, "0"), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1389 =#, Expr(:ref, :ENV, "GKSwstype") = :env), Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends/gr.jl"):1391 =#, Expr(:call, :pop!, :ENV, "GKSwstype")))))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Plots to MomentOpt:    
Expr(:call, Expr(:., :Sys, :(:isapple)))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Plots to MomentOpt:    
Expr(:||, Expr(:call, Expr(:., :Sys, :(:islinux))), Expr(:call, Expr(:., :Sys, :(:isbsd))))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Plots to MomentOpt:    
Expr(:call, Expr(:., :Sys, :(:iswindows)))
  ** incremental compilation may be broken for this module **

WARNING: eval from module Main to MomentOpt:    
Expr(:block, #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends.jl"):320 =#, Expr(:import, Expr(:., :Plots)), #= Symbol("/Users/74097/.julia/packages/Plots/ex9On/src/backends.jl"):321 =#, Expr(:export, :Plots))
  ** incremental compilation may be broken for this module **

Test Summary:           | Pass  Total
Running MomentOpt tests |    8      8
   Testing MomentOpt tests passed

fredrikekre · September 14, 2018, 9:30am

That is probably Plots fault.

Edit: See also https://github.com/JuliaLang/julia/issues/29059 and fix #29059, unnecessary warning in precompile for `eval` in `__init__` by JeffBezanson · Pull Request #29134 · JuliaLang/julia · GitHub