I’ve been using Julia off and on for a number of years and the one thing that keeps pushing me away from Julia is offline development. While Matlab has a core set of tools that are extensive, it lends itself to scientific programming and development on computers without an internet connection. My question is this, say I have a computer that’s completely disconnected from the internet and any internal networks (it’s a stand alone system).
How do I routinely add packages on a stand alone computer?
I can burn CD’s, yes, CD’s (remember those?). And bring files over 700MB at time. But, that’s the only way I’m allowed to migrate files over to this development computer. Does anyone here have experience with doing this? Or does anyone know what a typical process looks like?
One simple workflow improvement (not sure if it’s doable given your setup) is switching to USB. That will let you easily transfer 100 gb at a time which would prevent you needing multiple trips.
It’s true that I can’t use flash media. The rationale is that the security restrictions are in place that prevent this action. But, even if I could get files over (happily burn as many CD’s as it takes), how can packages be installed and loaded on completely offline systems? Any ideas?
The procedure is relatively straightforward - you burn your prepared .julia (or wherever your DEPOT_PATH is located at) to your CDs and copy them over. Then you set Pkg.offline(true) and you should be done.
Alternatively, you could mirror the General registry in your secure facility - though I’m sure talking to the folks at JuliaHub would be better for that kind of scenario.
The procedure is relatively straightforward - you burn your prepared .julia (or wherever your DEPOT_PATH is located at) to your CDs and copy them over. Then you set Pkg.offline(true) and you should be done.
Alternatively, you could mirror the General registry in your secure facility - though I’m sure talking to the folks at JuliaHub would be better for that kind of scenario.
So you need two machines?
One machine at home to download all the files. Then “burn” it on to a USB drive. Bring it to work.
And another machine at work which is air gaped. Download the files from your USB drive.
I can do that. I usually have used Pkg.add() in the past. But I guess I could go to a git page for that package and then see if I can download some files. Burn them to a CD. Then run it though an intermediate computer to scan for viruses. Once it passes go to the completely offline computer and try the Pkg.offline(true) trick above. I’ve never done it. How do I point Julia to the code on the CD so that it can load the package. When I do Pkg.add after I set offline to true, it will probably look for the package some place. Is there a way to set where that place is locally on the computer?
I’m guess that’s the Depot Path above. I need to read the documentation.
I know of a team at MIT Lincoln Lab that does this kind of offline deployment. I don’t think they ever wrote up their approach, but it boils down to create a local depot that mirrors what you want and if you want to you could in theory crawl the entirety of the public registry.
They do put everything on hard drives, but if I understand correctly those get destroyed when returning from operations
I have worked on air gapped HPC clusters. To be honest have never implemented Julia on such a cluster.
I would give some serious thought to using Artifactory
However they dont give examples of using Julia repositories with this setup. I would like to see if this works.
It’s the same thing Sukera already mentioned, one can choose the Julia depot to be located somewhere else than $HOME/.julia.
No, just do Pkg.add() as before, then copy the Julia depot. I think you also may want to prevent/interrupt the precompilation, to prevent wasted work in case the CPU is not the same on both machines. Here’s an example:
[nsajko@aceramd bin]$ JULIA_DEPOT_PATH=/home/nsajko/tmp/jl_depot ./julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.0-beta3 (2023-10-03)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.10) pkg> st
Installing known registries into `~/tmp/jl_depot`
Status `~/tmp/jl_depot/environments/v1.10/Project.toml` (empty project)
(@v1.10) pkg> up
Updating registry at `~/tmp/jl_depot/registries/General.toml`
No Changes to `~/tmp/jl_depot/environments/v1.10/Project.toml`
No Changes to `~/tmp/jl_depot/environments/v1.10/Manifest.toml`
[ Info: We haven't cleaned this depot up for a bit, running Pkg.gc()...
Active manifest files: 1 found
Active artifact files: 0 found
Active scratchspaces: 0 found
Deleted no artifacts, repos, packages or scratchspaces
(@v1.10) pkg> add MutableArithmetics
Resolving package versions...
Installed MutableArithmetics ─ v1.3.3
Updating `~/tmp/jl_depot/environments/v1.10/Project.toml`
[d8a4904e] + MutableArithmetics v1.3.3
Updating `~/tmp/jl_depot/environments/v1.10/Manifest.toml`
[d8a4904e] + MutableArithmetics v1.3.3
[0dad84c5] + ArgTools v1.1.1
[56f22d72] + Artifacts
[2a0f44e3] + Base64
[ade2ca70] + Dates
[f43a241f] + Downloads v1.6.0
[7b1f6079] + FileWatching
[b77e0a4c] + InteractiveUtils
[b27032c2] + LibCURL v0.6.4
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[ca575930] + NetworkOptions v1.2.0
[44cfe95a] + Pkg v1.10.0
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA v0.7.0
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[2f01184e] + SparseArrays v1.10.0
[fa267f1f] + TOML v1.0.3
[a4e569a6] + Tar v1.10.0
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
[e66e0078] + CompilerSupportLibraries_jll v1.0.5+1
[deac9b47] + LibCURL_jll v8.0.1+1
[e37daf67] + LibGit2_jll v1.6.4+0
[29816b5a] + LibSSH2_jll v1.11.0+1
[c8ffd9c3] + MbedTLS_jll v2.28.2+1
[14a3606d] + MozillaCACerts_jll v2023.1.10
[4536629a] + OpenBLAS_jll v0.3.23+2
[bea87d4a] + SuiteSparse_jll v7.2.0+1
[83775a58] + Zlib_jll v1.2.13+1
[8e850b90] + libblastrampoline_jll v5.8.0+1
[8e850ede] + nghttp2_jll v1.52.0+1
[3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
6 dependencies successfully precompiled in 9 seconds
(@v1.10) pkg> gc --all
Active manifest files: 1 found
Active artifact files: 0 found
Active scratchspaces: 0 found
Deleted no artifacts, repos, packages or scratchspaces
julia>
[nsajko@aceramd bin]$ bash
[nsajko@aceramd bin]$ cd /home/nsajko/tmp/jl
jl/ jl_depot/
[nsajko@aceramd bin]$ cd /home/nsajko/tmp/jl
jl/ jl_depot/
[nsajko@aceramd bin]$ cd /home/nsajko/tmp/jl_depot/
[nsajko@aceramd jl_depot]$ ls -l
total 0
drwxr-xr-x 1 nsajko nsajko 10 Oct 15 12:40 compiled
drwxr-xr-x 1 nsajko nsajko 10 Oct 15 12:39 environments
drwxr-xr-x 1 nsajko nsajko 94 Oct 15 12:41 logs
drwxr-xr-x 1 nsajko nsajko 36 Oct 15 12:40 packages
drwxr-xr-x 1 nsajko nsajko 52 Oct 15 12:39 registries
drwxr-xr-x 1 nsajko nsajko 72 Oct 15 12:39 scratchspaces
[nsajko@aceramd jl_depot]$ ls -l compiled/
total 0
drwxr-xr-x 1 nsajko nsajko 170 Oct 15 12:40 v1.10
[nsajko@aceramd jl_depot]$ rm -r compiled/ # save CD space
[nsajko@aceramd jl_depot]$ exit
exit
[nsajko@aceramd bin]$ # Now you would copy /home/nsajko/tmp/jl_depot to CDs
[nsajko@aceramd bin]$ # and carry on on the air-gapped machine.
[nsajko@aceramd bin]$ JULIA_DEPOT_PATH=/home/nsajko/tmp/jl_depot JULIA_PKG_OFFLINE=true ./julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.0-beta3 (2023-10-03)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.10) [offline] pkg> st
Status `~/tmp/jl_depot/environments/v1.10/Project.toml`
[d8a4904e] MutableArithmetics v1.3.3
Sorry for hijacking the thread but I cannot refrain.
My organization uses Artifactory to manage downloading external packages for many programming languages such as R and Python. I think that for many corporate users this is needed to get the language to be used at scale in their orgs.
If anyone knows how to make Artifactory work with pkg servers I would be immensely grateful.
I tried to find out more information about what Artifactory is and how it works, but I suspect the short answer is: You will have to file a support ticket with JFrog?
A bit slow on my response here, but thought this may be of interest to those interested in this thread.
I routinely deliver Julia packages into air gapped environments. I’ve extracted the depot-building code from the proprietary stuff into its own package:
FWIW I’m using this to build a Windows deliverable from my Mac, all thanks to how slick Julia’s Pkg server is. Please test it out and let me know how it goes!