Behavior of Julia package manager, deleting folders in registeries/General very frequently, unacceptable

General Usage Performance
,
1

Dear Julia developers and experts,

I am a scientific researcher and a regular Julia user. I often run Julia programs on HPC. During the development of my own package, I need to add other packages from time to time. Every time I add a package, the Pkg manager first delete everything in .julia/registeries/General and re-download it from the package server. This is OK for a laptop with fast internet connection. The problem is, the HPC login nodes I am using typically have slow internet connection and VERY SLOW disk access. Deleting folders in .julia/registeries/General costs several seconds on a laptop but can cost half an hour on the HPC login node. In addition, downloading also cost very long time. I have been doing nothing but deleting/re-downloading contents .julia/registeries/General in the last 72 hours !

I wish to learn how to avoid a complete update (not deleting .julia/registeries/General and re-downloading) but just install the package what I have instructed. Is there a reliable way to do so?

[lianyl@login1 ~]$ ll .julia/registries/General/
total 653
drwxr-xr-x  61 lianyl CSG   4096 Sep  1 09:33 F
drwxr-xr-x 319 lianyl CSG   4096 Sep  1 09:13 G
drwxr-xr-x 141 lianyl CSG   4096 Sep  1 09:14 H
drwxr-xr-x 230 lianyl CSG   4096 Sep  1 09:15 I
drwxr-xr-x 102 lianyl CSG   4096 Sep  1 09:15 J
drwxr-xr-x  79 lianyl CSG   4096 Sep  1 09:16 K
drwxr-xr-x 380 lianyl CSG   4096 Sep  1 09:17 L
-rw-r--r--   1 lianyl CSG   1071 Sep  1 09:17 LICENSE.md
drwxr-xr-x 482 lianyl CSG   4096 Sep  1 09:19 M
drwxr-xr-x 198 lianyl CSG   4096 Sep  1 09:20 N
drwxr-xr-x 159 lianyl CSG   4096 Sep  1 09:20 O
drwxr-xr-x 463 lianyl CSG   4096 Sep  1 09:22 P
drwxr-xr-x 108 lianyl CSG   4096 Sep  1 09:22 Q
drwxr-xr-x 272 lianyl CSG   4096 Sep  1 09:23 R
-rw-r--r--   1 lianyl CSG  13282 Sep  1 09:23 README.md
-rw-r--r--   1 lianyl CSG 567340 Sep  1 09:23 Registry.toml
drwxr-xr-x 702 lianyl CSG   4096 Sep  1 09:26 S
drwxr-xr-x 314 lianyl CSG   4096 Sep  1 09:27 T
drwxr-xr-x  84 lianyl CSG   4096 Sep  1 09:27 U
drwxr-xr-x  74 lianyl CSG   4096 Sep  1 09:27 V
drwxr-xr-x  78 lianyl CSG   4096 Sep  1 09:27 W
drwxr-xr-x 101 lianyl CSG   4096 Sep  1 09:28 X
drwxr-xr-x  36 lianyl CSG   4096 Sep  1 09:28 Y
drwxr-xr-x  33 lianyl CSG   4096 Sep  1 09:28 Z
[lianyl@login1 ~]$ ll .julia/registries/General/
total 20
drwxr-xr-x   6 lianyl CSG 4096 Sep  1 09:45 V
drwxr-xr-x  78 lianyl CSG 4096 Sep  1 09:27 W
drwxr-xr-x 101 lianyl CSG 4096 Sep  1 09:28 X
drwxr-xr-x  36 lianyl CSG 4096 Sep  1 09:28 Y
drwxr-xr-x  33 lianyl CSG 4096 Sep  1 09:28 Z
[lianyl@login1 ~]$ ll .julia/registries/General/
total 52
drwxr-xr-x 320 lianyl CSG 4096 Sep  1 09:47 A
drwxr-xr-x 279 lianyl CSG 4096 Sep  1 09:48 B
drwxr-xr-x 533 lianyl CSG 4096 Sep  1 09:50 C
-rw-r--r--   1 lianyl CSG 8015 Sep  1 09:50 CONTRIBUTING.md
drwxr-xr-x 353 lianyl CSG 4096 Sep  1 09:51 D
drwxr-xr-x 197 lianyl CSG 4096 Sep  1 09:52 E
drwxr-xr-x 294 lianyl CSG 4096 Sep  1 09:53 F
drwxr-xr-x 319 lianyl CSG 4096 Sep  1 09:54 G
drwxr-xr-x 141 lianyl CSG 4096 Sep  1 09:54 H
drwxr-xr-x 230 lianyl CSG 4096 Sep  1 09:55 I
drwxr-xr-x 102 lianyl CSG 4096 Sep  1 09:55 J
drwxr-xr-x  32 lianyl CSG 4096 Sep  1 09:56 K


[lianyl@login1 ~]$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.4 (2021-03-11)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.5.4
Commit 69fcb5745b (2021-03-11 19:13 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, broadwell)
2

I’m not sure about the Pkg side of this, but in the past I worked around similar problems by symlinking $HOME/.julia to a faster disk mount rather than using the very slow HPC home directory. Do you have some other mount points for faster disk available on your cluster? It probably wouldn’t need to be backed up.

4 Likes
3

You can try this trick here:

Or try on the latest 1.7 beta which by that topic may have a much faster download.

5 Likes
4

thanks!

5

I’ll try this now, thank you !

6

You can also do:

using Pkg
Pkg.offline()

At the beginning of your session.

help?> Pkg.offline()
  Pkg.offline(b::Bool=true)

  Enable (b=true) or disable (b=false) offline mode.

  In offline mode Pkg tries to do as much as possible without connecting to internet. For example, when adding a package Pkg only considers versions that
  are already downloaded in version resolution.

  To work in offline mode across Julia sessions you can set the environment variable JULIA_PKG_OFFLINE to "true".

  │ Julia 1.5
  │
  │  Pkg's offline mode requires Julia 1.5 or later.
4 Likes
7

In 1.7 one (actually two) files are used for the whole registry so (as been said) that should help.

4 Likes