Removing and adding packages

question
package

#1

If I manually remove a package directory from the repository I cannot normally install it. Example:

Step 1. I manually removed package directory from ~/.julia/packages/DataFrames/
Step 2. I try to install it using add DataFrames. I get an error:

(v1.0) pkg> add DataFrames
 Resolving package versions...
ERROR: path ~/.julia/packages/DataFrames/KUWId for package DataFrames no longer exists. Remove the package or `develop` it at a new path

Step 3. I remove the package (v1.0) pkg> rm DataFrames - all seems fine.

Step 4. I try to add the package (v1.0) pkg> add DataFrames and get the same error

Step 5. I try to add the package master: (v1.0) pkg> add DataFrames#master - now it works

Would it be possible to make (v1.0) pkg> add DataFrames work?


#2

Maybe https://github.com/JuliaLang/Pkg.jl/issues/913 (fix incoming with Julia 1.0.3) although that only applied to adding with a branch multiple times. You get this from just a regular add DataFrames?


#3

Yes - it seems that add DataFrames expects the folder to exist if it was manually removed, but e.g. add DataFrames#master creates the folder so it works.


#4

I can not reproduce this.


#5

Thank you for investigating into it. Here are detailed instructions how you can reproduce it.
Current version of DataFrames 0.15.2 installs in ~/.julia/packages/DataFrames/KUWId.

  1. Run add DataFrames#master
  2. Delete ~/.julia/packages/DataFrames/KUWId folder
  3. Run add DataFrames (without master) - you get the error
  4. Run rm DataFrames - seems to work
  5. Run add DataFrames (without master) - you get the same error still
  6. Run add DataFrames#master - now it is OK

And a second way (more problematic in my opinion):

  1. Run add DataFrames#master
  2. Run rm DataFrames - removes the package from the registry
  3. Delete ~/.julia/packages/DataFrames/KUWId folder - it should be OK (the package is not installed now)
  4. Run add DataFrames - you get an error, although you try to “freshly” install the package
  5. Now create ~/.julia/packages/DataFrames/KUWId folder (it is empty)
  6. Run add DataFrames - Julia tells you that DataFrames v0.15.2+ #master is installed (although I did not ask for master) and what is more ~/.julia/packages/DataFrames/KUWIdis still empty (although last command went through without a problem informing me that the installation was successfull)

Additionally I have the following question I stumbled when writing down the steps to reproduce the above:

  1. Run add DataFrames
  2. Delete file ~/.julia/packages/DataFrames/KUWId/src/DataFrames.jl
  3. Now the package is corrupded, so I try running add DataFrames to fix the corrupted state, but actually the folder ~/.julia/packages/DataFrames/KUWId does not get fixed

#6

FWIW, I observed a similar issue when ] up SomePackage. However, somewhat strangely it always worked the second time (exact same command, i.e. ] up SomePackage).


#7

In general it seems to me that the Package Manager probably caches something and then uses this information again when processing the commands.
Probably this is needed to ensure responsiveness (and this is great in comparison to an old package manager). But maybe:

  • there are some checks that are cheap and could be done
  • you could have a switch that forces the Package Manager to be in a “defensive” mode and check everything when running the command

#8

Ok, this is definitely https://github.com/JuliaLang/Pkg.jl/issues/913 then.


#9

Well, the files are write-protected, so you have to go to some length to actually remove files like this. But perhaps checking for the entry file makes sense, but since we can’t know if any other files have been removed I am not surehow much it helps to check for one file.


#10

On Windows they are not write-protected :frowning_face: (and I must admit with shame that sometimes I dabble with the repo if I have to make a quick-hack and I am to lazy to run develop).

A strategy could be e.g. to have a fingerprint of the state of the repository and checking if it has not changed (and if it changed then we know that the folder is corrupted). For most packages this should be quick (but if it happens to be slow then as I have proposed above there should be some option for turning on strict checking on demand).

Anyway - thank you for https://github.com/JuliaLang/Pkg.jl/issues/913 as hopefully this will solve most of the problems. What I am writing above is nice-to-have only, as this is normally not strictly required.


#11

There is actually another problem here. The problems described above is only partially fixed by #913.

Hmm, seems like a bug then, the chmod calls here https://github.com/JuliaLang/Pkg.jl/blob/2075b74621f7829a930a480fe4c3b8c8f8ac0924/src/Operations.jl#L65-L74 should set the file permission. Does that not happend? (This requires Julia 1.0.2 btw).


#12

I am on Julia 1.0.2 but I have admin privilleges on my laptop.