How to get dependency information for "to-be-added" packages?

New to Julia
,
1

Background: Like many, I have the issue that my work PC is behind a corporate firewall and I am not personally able to create a proxy (not an admin or in cybersec). Additionally, I have to submit approval requests to approve or update software, any 3rd-party dependencies that software has, and any 3rd party packages I wish to use in the software. Further, I have to get approval for the specific version of whatever package/dependency I need. Essentially, if the additional pieces don’t come “embedded” within the distribution of the approved software, I have to make a special request.

Questions: I’ve got several packages that I need add to my work Julia (1.3.1). Is there a way to get the list of dependencies of a package and determine whether they will require an update before I run Pkg.add() on my firewalled PC? For instance, on my personal (non-firewalled) PC (fresh Julia 1.3.1) I ran:

import Pkg
Pkg.add("MAT")

which prints the following to screen:

Updating registry at `C:\Users\Owner\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed Zstd_jll ─ v1.4.5+0
 Installed Compat ─── v3.10.0
  Updating `C:\Users\Owner\.julia\environments\v1.3\Project.toml`
  [23992714] + MAT v0.8.0
  Updating `C:\Users\Owner\.julia\environments\v1.3\Manifest.toml`
  [a74b3585] + Blosc v0.7.0
  [0b7ba130] + Blosc_jll v1.14.3+1
  [e1450e63] + BufferedStreams v1.0.0
  [944b1d66] + CodecZlib v0.7.0
  [34da2185] + Compat v3.10.0
  [f67ccb44] + HDF5 v0.13.2
  [0234f1f7] + HDF5_jll v1.10.5+5
  [5ced341a] + Lz4_jll v1.9.2+0
  [23992714] + MAT v0.8.0
  [3bb67fe8] + TranscodingStreams v0.9.5
  [83775a58] + Zlib_jll v1.2.11+10
  [3161d3a3] + Zstd_jll v1.4.5+0
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8bb1440f] + DelimitedFiles 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [a63ad114] + Mmap 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [1a1011a3] + SharedArrays 
  [6462fe0b] + Sockets 
  [2f01184e] + SparseArrays 
  [10745b16] + Statistics 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode
  1. Am I correct in thinking that Zstd_jll and Compat are packages that the command has acquired (via the web) and installed?
  2. Am I also correct in thinking that the items in the long list existed in my base installation, but had to download / update via the web?

If the answers are:

  • Yes, Yes:
    • I have to submit ~36 software approval requests, and have to request temporary proxy
  • Yes, already installed on PC just specify specific commit:
    • I only have to submit 2 software approval requests, and have to request temporary proxy
  • Already installed / existing (but not previously installed) in Julia installation, already installed:
    • I don’t have to submit any software approval requests, but still have to request temporary proxy

What I’m looking for is some way to make this determination either on my firewalled PC (which can’t run Pkg.add()) or by parsing the resultant Pkg.add text output on my personal PC so I can figure out my next steps.

2

Yes.

The list of versioned packages (like [f67ccb44] + HDF5 v0.13.2) are packages that you need but which version was already downloaded on your computer (from some previous Pkg command). So they didn’t need to be downloaded again. On another machine where you do Pkg.add they would need to get downloaded though.

The packages that do not have a version (like [2a0f44e3] + Base64) are “standard libraries” and comes bundled with Julia. They never need to be downloaded of the internet.

I am not sure what

means. You can for example look in the C:\Users\Owner\.julia\environments\v1.3\Manifest.toml file to see all packages that are required.

1 Like
3

Thanks for your help :slight_smile:

Regarding:

I think you answered the question with your answer above. Essentially, there doesn’t appear to be a Pkg.query_package_dependencies() function that I could run on my firewalled PC to figure out what dependencies I would need to install. The only way I see is to try to duplicate my Julia installation on my personal PC, run Pkg.add(), and parse the list (or .toml) according to your response above.

4

Hm, right now no, but in theory there is nothing preventing something like Pkg.add("Foo"; do_install=false) to exist that would do everything except downloading packages. Might be a good feature to add.

5

Right - and to clarify, this would still need access through the firewall? But I could at least request temporary proxy and then run that command (so that I can create approval requests).

6

I’ve not tried it yet, but could the new offline mode potentially be useful here?

For what it’s worth, one of the reasons for JuliaTeam is to help you deal with firewalls and governance policies. I’m always happy to chat more about this or hop on a call with you and/or your governance admins.

1 Like
7

It would one time need to download the “registry” (GitHub - JuliaRegistries/General: The official registry of general Julia packages) but after that it wouldn’t need the internet.

Like @mbauman says, there is also an offline mode (coming in 1.5) which will do it’s best to only use packages that are already installed on the system but you do have to get them there in the first place somehow.

8

Ah, yeah, I see the offline mode will just fail (and won’t tell you the list of all the dependencies that were missing, if any).

You might find the JuliaHub dependencies listing more helpful than what Pkg happens to update on your current machine. It lists both the direct and all indirect dependencies for every package, and explicitly separates out the Standard Libraries (which do not need to be downloaded).

1 Like