Pkg.status and Project.toml out of sync?

I have had the following issue several times and i dont’ understand. the output of ] st and what is written in my Project.toml does not correspond. How come? Synopsis:

➜  OsRo git:(master) ✗ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.1.0 (2019-01-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(v1.1) pkg> activate .

(OsRo) pkg> st
Project OsRo v0.1.0
    Status `~/.julia/dev/OsRo/Project.toml`
  [a93c6f00] + DataFrames v0.17.1
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.17.0
  [429524aa] + Optim v0.18.1
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 
    Status `~/.julia/dev/OsRo/Manifest.toml`
  [a93c6f00] + DataFrames v0.17.1
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.17.0
  [429524aa] + Optim v0.18.1
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 

shell> cat Project.toml
name = "OsRo"
uuid = "d06e37f3-36fe-5917-8577-8a051d4d43f5"
authors = ["florian oswald <florian.oswald@gmail.com>"]
version = "0.1.0"

[deps]
ArrayFire = "b19378d9-d87a-599a-927f-45f220a2c452"
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
CUDAdrv = "c5f51814-7f29-56b8-a69c-e4d8f6be1fde"
CUDAnative = "be33ccc6-a3ff-5ff2-a52e-74243cff1e17"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
DataFramesMeta = "1313f7d8-7da2-5740-9ea0-a2ca25f37964"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
GPUArrays = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
Optim = "429524aa-4258-5aef-a3af-852621145aeb"
ProfileView = "c46f51b8-102a-5cf2-8d2c-8597cb0e0da7"
ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
TimerOutputs = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test"]


shell> pwd
/Users/florian.oswald/.julia/dev/OsRo

you can see that a bunch of deps in the toml are missing from the st output. instantiate does not chane this:

(OsRo) pkg> instantiate
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`

(OsRo) pkg> st
Project OsRo v0.1.0
    Status `~/.julia/dev/OsRo/Project.toml`
  [a93c6f00] + DataFrames v0.17.1
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.17.0
  [429524aa] + Optim v0.18.1
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 
    Status `~/.julia/dev/OsRo/Manifest.toml`
  [a93c6f00] + DataFrames v0.17.1
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.17.0
  [429524aa] + Optim v0.18.1
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 

From the help:

pkg> help st
  status
  status [-p|--project]
  status [-m|--manifest]

  Show the status of the current environment. By default, the full contents of the project file is
  summarized, showing what version each package is on and how it has changed since the last git commit
  (if in a git repo), as well as any changes to manifest packages not already listed. In --project
  mode, the status of the project file is summarized. In --manifest mode the output also includes the
  dependencies of explicitly added packages.

where the key is how it has changed since the last git commit (if in a git repo). See also julia - How to prevent Pkg.jl to hold state beyond Project.toml and Manifest.toml - Stack Overflow

1 Like

ok, more to the point: How can i bring this project to the state of what is written in the Project.toml? i.e. how can i add the missing dependencies (short of manually adding them, whcih i want to avoid)

also, i’m not sure i understand the thing about how it has changed since the last commit. The relevant diff for that file from last commit to current is

diff --git a/Project.toml b/Project.toml
index ad616c1..902cb63 100644
--- a/Project.toml
+++ b/Project.toml
@@ -4,6 +4,7 @@ authors = ["florian oswald <florian.oswald@gmail.com>"]
 version = "0.1.0"
 
 [deps]
+ArrayFire = "b19378d9-d87a-599a-927f-45f220a2c452"
 BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
 CUDAdrv = "c5f51814-7f29-56b8-a69c-e4d8f6be1fde"
 CUDAnative = "be33ccc6-a3ff-5ff2-a52e-74243cff1e17"
@@ -19,3 +20,6 @@ TimerOutputs = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
 
 [extras]
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
+
+[targets]
+test = ["Test"]

so not really what st is showing?

They are not missing, they are just not displayed. You can commit the changes and then you will see the “absolute” status instead of the diff.

If you want to compare you should compare the diff of the Manifest.toml.

I’m sorry i’m slow to understand this. It seems inconsistent that st should not show all dependencies. here i verify that the repo is clean, all committed:

(v1.1) pkg> activate .

(OsRo) pkg> st
Project OsRo v0.1.0
    Status `~/.julia/dev/OsRo/Project.toml`
  [a93c6f00] + DataFrames v0.17.1
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.17.0
  [429524aa] + Optim v0.18.1
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 
  [8dfed614] + Test 
    Status `~/.julia/dev/OsRo/Manifest.toml`
  [a93c6f00] + DataFrames v0.17.1
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.17.0
  [429524aa] + Optim v0.18.1
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 
  [8dfed614] + Test 

shell> git status
On branch master
Your branch is ahead of 'origin/master' by 4 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

shell> cat Project.toml
name = "OsRo"
uuid = "d06e37f3-36fe-5917-8577-8a051d4d43f5"
authors = ["florian oswald <florian.oswald@gmail.com>"]
version = "0.1.0"

[deps]
ArrayFire = "b19378d9-d87a-599a-927f-45f220a2c452"
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
CUDAdrv = "c5f51814-7f29-56b8-a69c-e4d8f6be1fde"
CUDAnative = "be33ccc6-a3ff-5ff2-a52e-74243cff1e17"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
DataFramesMeta = "1313f7d8-7da2-5740-9ea0-a2ca25f37964"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
GPUArrays = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
Optim = "429524aa-4258-5aef-a3af-852621145aeb"
ProfileView = "c46f51b8-102a-5cf2-8d2c-8597cb0e0da7"
ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
TimerOutputs = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

This only worked after deleting the directory, and starting from scratch (forget about the build errors, I dont have a CUDA GPU on this machine…):

➜  ~ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.1.0 (2019-01-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(v1.1) pkg> dev git@github.com:floswald/OsRo.jl.git
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Cloning git-repo `git@github.com:floswald/OsRo.jl.git`
  Updating git-repo `git@github.com:floswald/OsRo.jl.git`
 Resolving package versions...
 Installed CUDAapi ──── v0.6.3
 Installed CUDAdrv ──── v3.0.0
 Installed ArrayFire ── v1.0.2
 Installed CUDAnative ─ v2.1.0
 Installed GPUArrays ── v0.7.0
 Installed LLVM ─────── v1.1.0
  Updating `~/.julia/environments/v1.1/Project.toml`
  [d06e37f3] + OsRo v0.1.0 [`~/.julia/dev/OsRo`]
  Updating `~/.julia/environments/v1.1/Manifest.toml`
  [b19378d9] + ArrayFire v1.0.2
  [3895d2a7] + CUDAapi v0.6.3
  [c5f51814] + CUDAdrv v3.0.0
  [be33ccc6] + CUDAnative v2.1.0
  [0c68f7d7] + GPUArrays v0.7.0
  [929cbde3] + LLVM v1.1.0
  [d06e37f3] + OsRo v0.1.0 [`~/.julia/dev/OsRo`]
  Building CUDAdrv ───→ `~/.julia/packages/CUDAdrv/3cR2F/deps/build.log`
┌ Error: Error building `CUDAdrv`: 
│ Could not find CUDA driver library.
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075
  Building LLVM ──────→ `~/.julia/packages/LLVM/tg8MX/deps/build.log`
  Building CUDAnative → `~/.julia/packages/CUDAnative/wU0tS/deps/build.log`
┌ Error: Error building `CUDAnative`: 
│ Dependent package CUDAdrv.jl has not been built successfully.
│ 
│ This is not a fatal error, but GPU functionality will be unavailable.
│ If you expected this to work, please open a thread on
│ https://discourse.julialang.org/c/domain/gpu
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075
  Building ArrayFire ─→ `~/.julia/packages/ArrayFire/QWNJQ/deps/build.log`
┌ Error: Error building `ArrayFire`: 
│ ┌ Warning: ArrayFire library not loaded:
│ │     either download a binary from the official site http://arrayfire.com/download
│ │     or you can build from source https://github.com/arrayfire/arrayfire
│ └ @ Main ~/.julia/packages/ArrayFire/QWNJQ/deps/build.jl:7
│ ERROR: LoadError: could not load library "libaf"
│ dlopen(libaf.dylib, 1): image not found
│ Stacktrace:
│  [1] #dlopen#3(::Bool, ::Function, ::String, ::UInt32) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Libdl/src/Libdl.jl:109
│  [2] dlopen at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
│  [3] top-level scope at /Users/florian.oswald/.julia/packages/ArrayFire/QWNJQ/deps/build.jl:5
│  [4] include at ./boot.jl:326 [inlined]
│  [5] include_relative(::Module, ::String) at ./loading.jl:1038
│  [6] include(::Module, ::String) at ./sysimg.jl:29
│  [7] include(::String) at ./client.jl:403
│  [8] top-level scope at none:0
│ in expression starting at /Users/florian.oswald/.julia/packages/ArrayFire/QWNJQ/deps/build.jl:4
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075

julia> cd(".julia/dev/OsRo")

(v1.1) pkg> activate .

(OsRo) pkg> st
Project OsRo v0.1.0
    Status `~/.julia/dev/OsRo/Project.toml`
  (no changes since last commit)

(OsRo) pkg> instantiate
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed QuadGK ──────────── v2.0.4
 Installed FixedPointNumbers ─ v0.6.1
 Installed FillArrays ──────── v0.6.1
 Installed DataFrames ──────── v0.18.2
 Installed Distributions ───── v0.20.0
  Updating `~/.julia/dev/OsRo/Project.toml`
  [b19378d9] + ArrayFire v1.0.2
  [6e4b80f9] + BenchmarkTools v0.4.2
  [c5f51814] + CUDAdrv v3.0.0
  [be33ccc6] + CUDAnative v2.1.0
  [a93c6f00] + DataFrames v0.18.2
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.20.0
  [0c68f7d7] + GPUArrays v0.7.0
  [429524aa] + Optim v0.18.1
  [c46f51b8] + ProfileView v0.4.0
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 
  Updating `~/.julia/dev/OsRo/Manifest.toml`
  [621f4979] + AbstractFFTs v0.4.1
  [79e6a3ab] + Adapt v0.4.2
  [7d9fca2a] + Arpack v0.3.1
  [b19378d9] + ArrayFire v1.0.2
  [6e4b80f9] + BenchmarkTools v0.4.2
  [9e28174c] + BinDeps v0.8.10
  [b99e7846] + BinaryProvider v0.5.4
  [e1450e63] + BufferedStreams v1.0.0
  [3895d2a7] + CUDAapi v0.6.3
  [c5f51814] + CUDAdrv v3.0.0
  [be33ccc6] + CUDAnative v2.1.0
  [159f3aea] + Cairo v0.5.6
  [49dc2e85] + Calculus v0.4.1
  [324d7699] + CategoricalArrays v0.5.2
  [3da002f7] + ColorTypes v0.7.5
  [5ae59095] + Colors v0.9.5
  [bbf7d656] + CommonSubexpressions v0.2.0
  [34da2185] + Compat v2.1.0
  [8f4d0f93] + Conda v1.2.0
  [a8cc5b0e] + Crayons v4.0.0
  [717857b8] + DSP v0.5.2
  [a93c6f00] + DataFrames v0.18.2
  [1313f7d8] + DataFramesMeta v0.4.1
  [864edb3b] + DataStructures v0.15.0
  [01453d9d] + DiffEqDiffTools v0.8.1
  [163ba53b] + DiffResults v0.0.4
  [b552c78f] + DiffRules v0.0.10
  [31c24e10] + Distributions v0.20.0
  [7a1cc6ca] + FFTW v0.2.4
  [5789e2e9] + FileIO v1.0.6
  [1a297f60] + FillArrays v0.6.1
  [53c48c17] + FixedPointNumbers v0.6.1
  [f6369f11] + ForwardDiff v0.10.3
  [0c68f7d7] + GPUArrays v0.7.0
  [a2bd30eb] + Graphics v0.4.0
  [4c0ca9eb] + Gtk v0.16.5
  [27996c0f] + GtkReactive v0.6.0
  [0862f596] + HTTPClient v0.2.1
  [d9be37ee] + Homebrew v0.7.1
  [8197267c] + IntervalSets v0.3.1
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [682c06a0] + JSON v0.20.0
  [929cbde3] + LLVM v1.1.0
  [b27032c2] + LibCURL v0.5.1
  [522f3ed2] + LibExpat v0.5.0
  [2ec943e9] + Libz v1.0.0
  [d3d80556] + LineSearches v7.0.1
  [e1d29d7a] + Missings v0.4.1
  [d41bc354] + NLSolversBase v7.3.1
  [77ba4419] + NaNMath v0.3.2
  [429524aa] + Optim v0.18.1
  [bac558e1] + OrderedCollections v1.1.0
  [90014a1f] + PDMats v0.9.7
  [d96e819e] + Parameters v0.10.3
  [f27b6e38] + Polynomials v0.5.2
  [2dfb63ee] + PooledArrays v0.5.1
  [85a6dd25] + PositiveFactorizations v0.2.2
  [c46f51b8] + ProfileView v0.4.0
  [92933f4c] + ProgressMeter v0.9.0
  [1fd47b50] + QuadGK v2.0.4
  [a223df75] + Reactive v0.8.3
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v0.5.2
  [79098fc4] + Rmath v0.5.0
  [d5f540fe] + RoundingIntegers v0.2.0
  [a2af1166] + SortingAlgorithms v0.3.1
  [276daf66] + SpecialFunctions v0.7.2
  [90137ffa] + StaticArrays v0.10.3
  [2913bbd2] + StatsBase v0.30.0
  [4c63d2b9] + StatsFuns v0.8.0
  [3783bdb8] + TableTraits v1.0.0
  [bd369af6] + Tables v0.2.4
  [a759f4b9] + TimerOutputs v0.5.0
  [30578b45] + URIParser v0.4.0
  [81def892] + VersionParsing v1.1.3
  [c17dfb99] + WinRPM v0.4.2
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8bb1440f] + DelimitedFiles 
  [8ba89e20] + Distributed 
  [9fa8497b] + Future 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [a63ad114] + Mmap 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [9abbd945] + Profile 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [1a1011a3] + SharedArrays 
  [6462fe0b] + Sockets 
  [2f01184e] + SparseArrays 
  [10745b16] + Statistics 
  [4607b0f0] + SuiteSparse 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 

(OsRo) pkg> st
Project OsRo v0.1.0
    Status `~/.julia/dev/OsRo/Project.toml`
  [b19378d9] + ArrayFire v1.0.2
  [6e4b80f9] + BenchmarkTools v0.4.2
  [c5f51814] + CUDAdrv v3.0.0
  [be33ccc6] + CUDAnative v2.1.0
  [a93c6f00] + DataFrames v0.18.2
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.20.0
  [0c68f7d7] + GPUArrays v0.7.0
  [429524aa] + Optim v0.18.1
  [c46f51b8] + ProfileView v0.4.0
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 
    Status `~/.julia/dev/OsRo/Manifest.toml`
  [b19378d9] + ArrayFire v1.0.2
  [6e4b80f9] + BenchmarkTools v0.4.2
  [c5f51814] + CUDAdrv v3.0.0
  [be33ccc6] + CUDAnative v2.1.0
  [a93c6f00] + DataFrames v0.18.2
  [1313f7d8] + DataFramesMeta v0.4.1
  [31c24e10] + Distributions v0.20.0
  [0c68f7d7] + GPUArrays v0.7.0
  [429524aa] + Optim v0.18.1
  [c46f51b8] + ProfileView v0.4.0
  [92933f4c] + ProgressMeter v0.9.0
  [a759f4b9] + TimerOutputs v0.5.0
  [9a3f8284] + Random 

shell> cat Project.toml
name = "OsRo"
uuid = "d06e37f3-36fe-5917-8577-8a051d4d43f5"
authors = ["florian oswald <florian.oswald@gmail.com>"]
version = "0.1.0"

[deps]
ArrayFire = "b19378d9-d87a-599a-927f-45f220a2c452"
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
CUDAdrv = "c5f51814-7f29-56b8-a69c-e4d8f6be1fde"
CUDAnative = "be33ccc6-a3ff-5ff2-a52e-74243cff1e17"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
DataFramesMeta = "1313f7d8-7da2-5740-9ea0-a2ca25f37964"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
GPUArrays = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
Optim = "429524aa-4258-5aef-a3af-852621145aeb"
ProfileView = "c46f51b8-102a-5cf2-8d2c-8597cb0e0da7"
ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
TimerOutputs = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test"]

Inconsistent how? When in a git repo that has changed compared to the files in the last commit status shows the diff.

The reason your git repo looks clean but status still shows a diff is that you probably have your Manifest.toml file .gitignored, so compared to the (non-existing) Manifest.toml file in HEAD this is the diff.

Ok yes the manifest is ignored by default from pkgTemplates I think. Still need to understand better. Thanks!