Package install on (offline) Cluster - Build needs internet

Specific Domains Julia at Scale
,
1

Hey everyone,

I’m trying to use julia 1.0 on our cluster.
It is not connected to the internet (login via login-server)
so I can’t use the standard procedure to install the packages I need.

I read here that one option should
be able to set up the .julia folder on a separate computer and copy it to the cluster.
(But maybe that was only true for pre-1.0?)

I tried it but got the following error

ERROR: LoadError: InitError: /home/isensee/.julia/packages/SpecialFunctions/fvheQ/deps/usr/lib/libopenspecfun.so cannot be opened, 
Please re-run Pkg.build("SpecialFunctions"), and restart Julia.

When I try to follow the order I get

(v1.0) pkg> build SpecialFunctions
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
┌ Warning: Some registries failed to update:
│     — /home/isensee/.julia/registries/General — failed to fetch from repo
└ @ Pkg.API /usr/lfpn/SOURCES.ORIGINAL/julia/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:144
  Building SpecialFunctions → `~/.julia/packages/SpecialFunctions/fvheQ/deps/build.log`
┌ Error: Error building `SpecialFunctions`: 
│ [ Info: Downloading https://github.com/JuliaMath/OpenspecfunBuilder/releases/download/v0.5.3-3/Openspecfun.v0.5.3.x86_64-linux-gnu-gcc4.tar.gz to /home/isensee/.julia/packages/SpecialFunctions/fvheQ/deps/usr/downloads/Openspecfun.v0.5.3.x86_64-linux-gnu-gcc4.tar.gz...
│ ERROR: LoadError: Could not download https://github.com/JuliaMath/OpenspecfunBuilder/releases/download/v0.5.3-3/Openspecfun.v0.5.3.x86_64-linux-gnu-gcc4.tar.gz to /home/isensee/.julia/packages/SpecialFunctions/fvheQ/deps/usr/downloads/Openspecfun.v0.5.3.x86_64-linux-gnu
-gcc4.tar.gz:
│ ErrorException("")
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] macro expansion at ./logging.jl:313 [inlined]
│  [3] #download#89(::Bool, ::Function, ::String, ::String) at /home/isensee/.julia/packages/BinaryProvider/4F5Hq/src/PlatformEngines.jl:487
│  [4] #download at ./none:0 [inlined]
│  [5] #download_verify#90(::Bool, ::Bool, ::Bool, ::Function, ::String, ::String, ::String) at /home/isensee/.julia/packages/BinaryProvider/4F5Hq/src/PlatformEngines.jl:567
│  [6] #download_verify at ./none:0 [inlined]
│  [7] #install#129(::Prefix, ::String, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at /home/isensee/.julia/packages/BinaryProvider/4F5Hq/src/Prefix.jl:314
│  [8] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{Prefix,Bool,Bool}}, ::typeof(install), ::String, ::String) at ./none:0
│  [9] top-level scope at /home/isensee/.julia/packages/SpecialFunctions/fvheQ/deps/build.jl:142
│  [10] include at ./boot.jl:317 [inlined]
│  [11] include_relative(::Module, ::String) at ./loading.jl:1038
│  [12] include(::Module, ::String) at ./sysimg.jl:29
│  [13] include(::String) at ./client.jl:388
│  [14] top-level scope at none:0
│ in expression starting at /home/isensee/.julia/packages/SpecialFunctions/fvheQ/deps/build.jl:139
│ [17:14:15] 
│ [17:14:15] curl: (7) Couldn't connect to server
└ @ Pkg.Operations /usr/lfpn/SOURCES.ORIGINAL/julia/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1068

So it seems that a Pkg.build from a different computer is not compatible and rebuilding fails due
to missing internet connection.
Is there any way out?

2

@JonasIsensee I have been getting the ame error messages regardign the need to rebuild SpecialFunctions. I have been workign with the 1.1.0rc2 in a Singularity container environment.
I think we need a discussion on installing Julia and installing packages on isolated systems.

3

I know this does not add to the discussion.
But are you SURE your the login server on yoru cluster cannot download packages?
If there is a proxy server you can set the HTTP_PROXY environment variable and probably get it ti to work.

4

You are right.
I can download the packages (] add MyPackage)
from the login server.

Doing ] build SpecialFunctions on the login server
does not solve my problem though as
the cluster node still shows the same error as before.

I am trying to do using MyPackage on the login-server now
( which we’re probably not supposed to do, as it’s a rather weak machine only meant for forwarding ssh sessions)
but I’m not even sure if that’d make a difference

EDIT: It does not help. I still get

julia> using TimeseriesPrediction
[ Info: Recompiling stale cache file /home/isensee/.julia/compiled/v1.0/TimeseriesPrediction/3Yac0.ji for TimeseriesPrediction [f218859d-9706-56aa-9ebf-1fa4ed7b8020]
ERROR: 
LoadError: InitError: /home/isensee/.julia/packages/SpecialFunctions/fvheQ/deps/usr/lib/libopenspecfun.so cannot be opened,
Please re-run Pkg.build("SpecialFunctions"), and restart Julia.

I don’t know of any proxy server but sent an email to see if one exists.

5

It’s working now.

I talked to our admin and he didn’t know of a proper solution either
so in the end he temporarily activated the internet access until
all packages were installed.

As long as I don’t have to update anything I’ll be fine for the current project
but this is really not a satisfactory solution.

6

Did you ever find a solution for this? I’m running into the exact same issue and getting internet access is not an option for me.

7

Hi @mopg,
sadly no.
The IT guy has probably forgotten that he wanted to turn off the internet access again.
So I’m still installing packages the normal way.

8

It seems too late for a response but since this question comes up in the searches I post here.
I created a Julia program that mirrors the Julia repository for offline clusters. It is not an advanced code but it solved my problems. nativeJuliaMirror.jl