Offline installation of Julia packages

package
installation

#1

Hello,

I need to install a bunch of packages on an offline computer ( it must be disconnected ).

I didn’t found any guide that provide a clean way for installing packages offline and maybe someone could help me with that.

I tried this :

  1. For each packages, download its github repo ( not with git clone ) and store it somewhere at local/path/pkg_n
  2. Add the local/path/pkg_n to these packages via Pkg

But when adding these local path, Pkg return this :

(v1.1) pkg> add local\path\pkg_1
Updating registry at ~\.julia\registries\General
Updating git-repo https://github.com/JuliaRegistries/General.git
Cloning git-repo local\path\pkg_1\
ERROR: Git repository not found at ‘local\path\pkg_1’

So the local path must be a git repo.

Then I tried with dev since it will create automatically a repo and update julia environnement Manifest.toml and Project.toml files.

For packages with a lot of dependencies, I could do this on all the packages in the dependency graph that are not already installed on hope for the best, but I haven’t tried yet.

Does anyone have any idea on how to do this properly ?

Thank you in advance !


#2

I think currently there is no way of doing that, see


However, I think you can try installing in another computer, and then copy the ~/.julia directory to the offline computer. Sometimes it gives problem by differences in the systems, but it is the simpler way. I suggest you to test it.


#3

Well with this solution the problem is that I can only use packages built on a windows OS, and since the offline computer is using debian I think it is going to be tricky .

I might get around that by using Pkg.develop and configuring myself the dependency like I would do for a private package.

It is going to be cumbersome but I guess I don’t have another choice.


#4

Use a docker container.


#5

@gdestouet You might look at Julia Team https://juliacomputing.com/products/juliateam.html


#6

Thank you for your response.
So if ever want to add or upgrade a package, I will have to build a docker image and reinstall it ? It seems a bit cumbersome at the beginning but I will give it a try.


#7

Did you do this using ‘git clone’ ? Please give us an example of the package you wish to install and the command you used to create the clone. Thanks!


#8

You can automate it. Also, offline environments are already cumbersome, the best you can do is to come up with a workflow that mitigates this to some extent.


#9

For a simple test I tried with Unitful.jl, I wasn’t using git clone for this but only the decompressed archive.

With git clone it works well when using Pkg.add, I am going to do this for any packages I need.

I will do this with a package with many dependencies and come back with what I did.


#10

Unfortunately I can’t use Docker Images and I would prefer to build packages on the offline machine, so here is what I did :

Let say that I have a project on a computer connected to internet that I want to export and install on an offline computer along with its dependencies ( outside Julia’s stdlib ).

For this project I have the canonical structure :

path/to/Project_test
│   Manifest.toml
│   Project.toml
└───src/
        Project_Test.jl
        *.jl

I make sure that all dependencies of Project_Test are installed and present in Manifest.toml and that it is running properly. And that both computers have the same project with the same dependencies.

Instead of copying all the packages from the default depot path (.julia/packages/...) to julia depot path on the offline computer. The following function ( a flattened and pruned version of Pkg.Types.add_or_develop and Pkg.Operates.add_or_develop combined ) will download all the packages needed and store it on the external_device at path_to_external . It will keep the structure used in the default depot path ( version-dependent ).It will also copy the registry to avoid Pkg.Types.manifest_info to call for a remote clone of the default registry while installing packages on the offline computer.

import Pkg.Types: PackageSpec,GitRepo,Context,project_deps_resolve!,registry_resolve!,stdlib_resolve!,ensure_resolved,manifest_info,write_env,update_registries,pkgerror,printpkgstyle
import Pkg.activate
import Pkg.Operations:resolve_versions!,version_data!,install_archive
import Pkg.BinaryProvider
using UUIDs
import LibGit2

function export_packages(path_to_project,path_to_external)
    activate(path_to_project)
    ctx=Context()
    project = ctx.env.project

    pkgs = [ PackageSpec(k,v) for (k,v) in project.deps ]

    project_deps_resolve!(ctx.env,pkgs)
    registry_resolve!(ctx.env,pkgs)
    stdlib_resolve!(ctx,pkgs)
    ensure_resolved(ctx.env,pkgs)
    for pkg in pkgs
        ctx.env.project.deps[pkg.name] = pkg.uuid
    end

    for (name::String, uuid::UUID) in ctx.env.project.deps
        entry = manifest_info(ctx.env, uuid)
        entry !== nothing && entry.version !== nothing || continue
        version = VersionNumber(entry.version)
        for pkg in pkgs
            pkg.uuid == uuid && version ∈ pkg.version || continue
            pkg.version = version
        end
    end

    resolve_versions!(ctx, pkgs)
    hashes, urls = version_data!(ctx, pkgs)

    for pkg in pkgs
        pkg.uuid in keys(ctx.stdlibs) && continue
        pkg.repo = GitRepo(urls[pkg.uuid][1],nothing,nothing)
    end

    BinaryProvider.probe_platform_engines!()
    pkgs_to_install = Tuple{PackageSpec, String}[]
    for pkg in pkgs
        pkg.uuid in keys(ctx.stdlibs) && continue
        path = joinpath(path_to_external,pkg.name,Base.version_slug(pkg.uuid,hashes[pkg.uuid]))
        push!(pkgs_to_install, (pkg, path))
    end

    widths = [textwidth(pkg.name) for (pkg, _) in pkgs_to_install]
    max_name = length(widths) == 0 ? 0 : maximum(widths)


    results = Channel(length(pkgs));
    i=1
    for (pkg, path) in pkgs_to_install
        print("$i over $(length(pkgs_to_install)) downloaded packages\r")
        try
            success = install_archive(urls[pkg.uuid], hashes[pkg.uuid], path)
            if ctx.use_only_tarballs_for_downloads && !success
                pkgerror("failed to get tarball from $(urls[pkg.uuid])")
            end
            put!(results, (pkg, success, path))
        catch err
            put!(results, (pkg, err, catch_backtrace()))
        end
        i+=1
    end

    missed_packages = Tuple{PackageSpec, String}[]
    for i in 1:length(pkgs_to_install)
        pkg, exc_or_success, bt_or_path = take!(results)
        exc_or_success isa Exception && pkgerror("Error when installing package $(pkg.name):\n",
                                                 sprint(Base.showerror, exc_or_success, bt_or_path))
        success, path = exc_or_success, bt_or_path
        if success
            vstr = pkg.version != nothing ? "v$(pkg.version)" : "[$h]"
            printpkgstyle(ctx, :Installed, string(rpad(pkg.name * " ", max_name + 2, "─"), " ", vstr))
        else
            push!(missed_packages, (pkg, path))
        end
    end

    for (pkg, path) in missed_packages
        uuid = pkg.uuid
        if !ctx.preview
            install_git(ctx, pkg.uuid, pkg.name, hashes[uuid], urls[uuid], pkg.version::VersionNumber, path)
        end
        vstr = pkg.version != nothing ? "v$(pkg.version)" : "[$h]"
        @info "Installed $(rpad(pkg.name * " ", max_name + 2, "─")) $vstr"
    end

end

Once the packages are stored on the external device. I used the following script to install and build them on the offline computer.

import Pkg:activate,depots1
import Pkg.Types:Context
import Pkg.Operations:build_versions

function add_local_packagesv11(pkgs_path,project_path)
    activate(project_path)
    ctx=Context()
    uuids_to_build=collect(keys(ctx.env.manifest))
    files=Set(readdir(pkgs_path))
    registry_path = joinpath(depots1(),"registries")
    Base.cp(joinpath(pkgs_path,pop!(files,"registries")),registry_path,force=true)
    default_path=joinpath(depots1(),"packages")
    for file in files
        path=joinpath(pkgs_path,file)
        if isdir(path)
            Base.cp(path,joinpath(default_path,file),force=true)
        end
    end
    build_versions(ctx,uuids_to_build)
end

project_path = "/path/to/Project_Test"
pkgs_path= "/path/to/external_device/pkgs"
add_local_packagesv11(pkgs_path,project_path)

Works well if the packages used doesn’t require internet to build, in which case you will have to configure the deps/build.jl scripts.
For example, if I want to build DSP on the offline computer I will have to modify its build.jl and download the specific FFTW library it needs.
Whereas, Conda builds well offline.

Tested with a project with ~40 dependencies on Julia 1.1.0.