V1.0 Pkg - Can't copy packages

[Glen1]

I’m so frustrated.
We’ve implemented our whole production environment in Julia and working really well, but can’t get v1.0 packages working.
We are behind a firewall, so can’t add/download packages, so I currently have the following:

• v5.2
• compile and build packages and dependencies on an external server
• copy compiled packages to production server
• manually change dep.jl files and all compile paths in packages
• Change the relevant Julia run-time path variables for new directory
  – works perfectly

How on earth do we do this in v1.x.

• The documentation doesn’t seem to covert this, or is unclear
• Changing DEPOT_PATH doesn’t really help

Failed Scenario:

• I’ve compiled GLM on another server and copied ~/.julia/* to production directory
• changed DEPOT_PATH to include new path
• I’d expect this to work… or at least be intuitive on where to make changes
• If I look in the copied path for GLM, I find it in:
  ./packages/GLM
  ./registries/General/G/GLM
  ./compiled/v1.0/GLM
  and if I try to use GLM, I get:
  ERROR: IOError: stat: permission denied (EACCES)
  Stacktrace:
  [1] stat(::String) at ./stat.jl:68
  [2] samefile at ./stat.jl:311 [inlined]
  [3] macro expansion at ./logging.jl:305 [inlined]
  [4] macro expansion at ./logging.jl:318 [inlined]
  [5] stale_cachefile(::String, ::String) at ./loading.jl:1330
  [6] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:693
  [7] _require(::Base.PkgId) at ./loading.jl:937
  [8] require(::Base.PkgId) at ./loading.jl:858
  [9] macro expansion at ./logging.jl:309 [inlined]
  [10] require(::Module, ::Symbol) at ./loading.jl:840

… I’m presuming that somehow Julia is looking in a wrong path, but where?
I thought that maybe the underlying optimizer is looking for a C library, but the message doesn’t seem to reflect that…

The same is true for DataFrames, which, as far as I know, wouldn’t use any dependent C libraries

Any help would be great.

[crbinz]

Here are a couple relevant topics that might have some helpful information:

How does one set up a centralized Julia installation?

I would like to start a thread about best practice for how to set up a centralized Julia installation in a cluster environment or local institution network. Hopefully, this will lead to a tutorial like write-up in the end. I want to focus on Julia 1.0 and, thus, Pkg(3). How does one set up a system wide Julia 1.0 installation, that is julia itself as well as commonly used packages? Requirements: Users should still be able to install packages not present in the central directories. As package…

How to create a system administrator depot?

This is from the Pkg section of the Julia doc about the DEPOT_PATH: How do I set up such a system administrator depot? Basically, what I want is to install a couple of packages system wide such that users don’t have to install them by themself. (This is in the context of How does one set up a centralized Julia installation?)

[00vareladavid]

That error in particular seems like either DEPOT_PATH was not set correctly or perhaps the files were copied over with incorrect permissions.

Also what is your workflow on the production server? Do you need something centralized with multiple users or only a single user?

[kristoffer.carlsson]

What you do should work but something seems to be going wrong when the already precompiled files in .julia/compiled are getting loaded. Like @00vareladavid said, please check that the permission of the files are correct. To check if the DEPOT_PATH is correct, you can start julia in the production environment and do a import Pkg; Pkg.status(). GLM should be listed and it should not warn about being uninstalled if the DEPOT_PATH is set correctly.

Perhaps you can try not copying over the precompilation file (the files in compiled) and let them be regenerated on the production server and see what happens in that case.

[Glen1]

Thanks — permissions were fine — seemed to be the already compiled/cached packages from the previous system — once files under /…/compiled/v1.0/* were deleted, worked like a charm

[Glen1]

Amazing!!! THANK YOU

This worked:

1. copy .julia to <new_.julia_path> (and different system)
2. chmod -R 777 <new_.julia_path>
3. rm -Rf <new_.julia_path>/compiled/v1.0/*
4. push!(LOAD_PATH,“<new_.julia_path>”)
5. empty!(DEPOT_PATH)
   push!(DEPOT_PATH,“<new_.julia_path>”)
6. using Pkg
   Pkg.status()
7. using GLM, DataFrames, CSV

Glen

[xiaodai]

Wonder if you can help me here.

I followed the instructions but I get this

using DataFrames
ERROR: ArgumentError: Package DataFrames not found in current path:
- Run `import Pkg; Pkg.add("DataFrames")` to install the DataFrames package.

Both my LOAD_PATH and DEPOT_PATH contains the new_.julia_path. But I copied the folders from Windows to Linux.

Edit

I realised that the issue was that I copied v1 library to v1.1. So I had to just copy the /environments/v1.0 to /environments/v1.1 and it worked!!!

[jwu]

I have to manually modify the environment/Manifest.toml for development packages.

[kristoffer.carlsson]

I don’t understand what that means.