Pkg.add("FourierFlows") takes long time -- did I do something wrong?

New to Julia
1

I have committed the first tagged version for FourierFlows.jl #12685 and it was merged successfully.

However, when I do Pkg.add("FourierFlows") on a brand new clean installation I get

julia> Pkg.init()
INFO: Initializing package repository /Users/navid/.julia/v0.6
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl

julia> @time Pkg.add("FourierFlows")
INFO: Cloning cache of BinDeps from https://github.com/JuliaLang/BinDeps.jl.git
INFO: Cloning cache of BufferedStreams from https://github.com/BioJulia/BufferedStreams.jl.git
INFO: Cloning cache of Compat from https://github.com/JuliaLang/Compat.jl.git
INFO: Cloning cache of DataStructures from https://github.com/JuliaCollections/DataStructures.jl.git
INFO: Cloning cache of FileIO from https://github.com/JuliaIO/FileIO.jl.git
INFO: Cloning cache of FourierFlows from https://github.com/FourierFlows/FourierFlows.jl.git

Then it stays at this point for ~40 min before it proceeds with

INFO: Cloning cache of JLD2 from https://github.com/simonster/JLD2.jl.git
INFO: Cloning cache of Libz from https://github.com/BioJulia/Libz.jl.git
INFO: Cloning cache of SpecialFunctions from https://github.com/JuliaMath/SpecialFunctions.jl.git
INFO: Installing BufferedStreams v0.3.3
INFO: Installing DataStructures v0.7.4
INFO: Installing FileIO v0.6.1
INFO: Installing FourierFlows v0.0.1
INFO: Installing JLD2 v0.0.5
INFO: Installing Libz v0.2.4
INFO: Installing SpecialFunctions v0.3.7
INFO: Building SpecialFunctions
INFO: Package database updated
INFO: METADATA is out-of-date — you may not have the latest version of FourierFlows
INFO: Use `Pkg.update()` to get the latest versions of your packages
2756.170447 seconds (5.68 M allocations: 388.634 MiB, 0.01% gc time)

Isn’t this strange? Why Pkg.add() needs 40 min? Did I do something wrong when I registered the package?

For the record:

julia> versioninfo()
Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin17.3.0)
  CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

By the way on a linux machine Pkg.add("FourierFlows") takes ~20sec – much more reasonable.

1 Like
2

Let me add some additional information that might be relevant.

Initially the repository was in my personal github account: navidcy
After I submitted the PR in METADATA.jl for registering v0.0.1 I created an organization FourierFlows and moved the repository there. I have updated the repositories url in METADATA.jl (PR #12762).

Is this, by any chance, responsible for the long time Pkg.add() is taking?

3

No, that wouldn’t do anything.

Are you installing from a different computer? Behind a proxy?

4

I’m not behind a proxy.
I’ve tried from various internet connections and also @glwagner tried from his own – same problem. Hopefully when we submit the v0.0.2 release this issue will be resolved. But any suggestions are welcome!

5

Maybe it’s related to having __precompile__() on every page? I’ve never seen that before so I actually have no idea what that would do. A single __precompile__() is all that’s necessary, and at least something to try first.

6

Yes, I’ve already fixed that in the latest version of FourierFlows – thanks.

7

For the record: I can reproduce the problem in my mac.

8

This is installing FourierFlows on Pkg3:

https://giphy.com/gifs/l49JE1ZSFrOgRW5e8/fullscreen

It would be interesting if you could try it with Pkg3 (https://github.com/JuliaLang/Pkg3.jl) on the computer where the extreme slowness is ocurring.

9

@kristoffer.carlsson : Indeed. Pkg3 installed FourierFlows in ~20sec.

10

So still this problem comes up erratically…
Pkg.add("FourierFlows") gets stuck at:

julia> Pkg.add("FourierFlows")
INFO: Cloning cache of FourierFlows from https://github.com/FourierFlows/FourierFlows.jl.git

Mind blowing. I never had this problem with any other packages other than FourierFlows.

11

Just tried it here on 0.6.2. It’s been a good 20 minutes and it’s still not done. The julia process is still receiving bytes though; 480 MB downloaded so far. Could it be that FourierFlows has large objects in its git history? Pkg3 doesn’t download the whole git history, so that’s a big difference.

Edit: 55 minutes total on my system :slight_smile:.

12

I hadn’t read this earlier. Can reproduce this too; about 40 seconds on my machine (though on a different network as well). But still, on Linux, df -sh .git reports 848 MB, which seems excessive.

13

Could be https://github.com/libgit2/libgit2/issues/4464 as well (although I don’t know how Pkg2 add is implemented). Note that that issue is closed as a duplicate of an issue that is still open.

14

@tkoolen: So, have I made something wrong in the repository? Perhaps at some point in its history there was an accidental push of large files. Is there a way to clean it?

15

The standard LibGit2 clone is fast:

julia> @time LibGit2.clone("https://github.com/FourierFlows/FourierFlows.jl", "FourierFlows")
  3.818350 seconds (466.33 k allocations: 27.295 MiB, 0.12% gc time)
LibGit2.GitRepo("/Users/kristoffer/FourierFlows.jl/FourierFlows")
16

What does Pkg.add use?

17

Another piece of the puzzle: with the standard LibGit2 clone or when cloning from the command line, the .git directory ends up being ~1.7 MB, not 848 MB as with Pkg.add.

1 Like
18

@tkoolen: Indeed that was the problem, the repository was clogged up from some accidental commit of many .png files. Now everything is sorted out.

2 Likes