Building a Dockerfile with packages

New to Julia
,
1

I want to build a docker image with Julia packages ready to go.
What are best practices for that?

Here’s what I got so far:

FROM julia

RUN julia install.jl

where install.jl is:

using Pkg
packages = ["JSON", "CSV"]
Pkg.add(packages)
using CSV, JSON

Is there any way to precompile an array of packages?

From the docs I thought that Pkg.build accepts array of packages (same as Pkg.add) but it throws an error:

Pkg.build(packages)
ERROR: MethodError: no method matching Pkg.Types.PackageSpec(::Array{String,1})
Closest candidates are:
  Pkg.Types.PackageSpec(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Types.jl:91
  Pkg.Types.PackageSpec(; name, uuid, version, tree_hash, repo, path, pinned, mode) at util.jl:742
  Pkg.Types.PackageSpec(::Union{Nothing, String}, ::Union{Nothing, Base.UUID}, ::Union{Pkg.Types.UpgradeLevel, VersionNumber, Pkg.Types.VersionSpec}, ::Union{Nothing, Base.SHA1}, ::Pkg.Types.GitRepo, ::Union{Nothing, String}, ::Bool, ::Pkg.Types.PackageMode) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Types.jl:91
  ...
Stacktrace:
 [1] iterate at .\generator.jl:47 [inlined]
 [2] collect(::Base.Generator{Tuple{Array{String,1}},Type{Pkg.Types.PackageSpec}}) at .\array.jl:665
 [3] #build#105 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:677 [inlined]
 [4] build(::Array{String,1}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:677
 [5] top-level scope at REPL[4]:1

Although if I hardcode the array it works:

Pkg.add(["CSV", "JSON"])

But I want to avoid typing modules list twice. How can I do that?

2

You can first add all the packages you need, and then run precompile - it should precompile all installed packages:

using Pkg
pkg"add CSV"
pkg"add JSON"
pkg"precompile"

At least that’s how I do it.

1 Like
3

This is also how I do it.

However, for the sake of completeness, I would also like to add another method, which works especially well if you’re starting from a fresh Julia install:

  1. prepare your environment on your local machine by manually Pkg.adding all packages you want
  2. copy the corresponding Project.toml (and possibly the associated Manifest) to ~/.julia/environments/v1.x/ in the Docker file
  3. run pkg"instantiate" within the docker image to install all packages
  4. as mentioned by @aplavin, run pkg"precompile" in the docker image to precompile everything

Using a Project.toml and Manifest.toml instead of a list of Pkg.add commands might bring you a few benefits:

  • thanks to the manifest, you always get the exact same versions of all packages
  • if you use unregistered packages, their url is stored in the manifest (this means that unregistered packages are no special case, whereas the Pkg.add invocation for them is a bit more complex than for registered packages)
6 Likes
4

This thread might be of interest

5

Also is this any use to you?

6

This package creates ridiculously long Dockerfile with 2.3gb in size. It even downloads tex live environment. Huge waste of space, besides running all sort of tests… Not seemed practical to me.

7

Rather than directly copying files into ~/.julia/environments/v1.x/ (and bypassing Pkg) , it may be better to copy *.toml to a (empty) dir /foo and
execute `julia --project=/foo -e “using Pkg; Pkg.instantiate(); Pkg.precompile()”

1 Like
8

I like @o314’s approach. For completeness, the process is as follows:

  1. Create a new Julia environment and add the required packages.
mkdir Foo
cd Foo && julia -e 'using Pkg; Pkg.activate("."); Pkg.add(["HTTP", "DataFrames"]);'
  1. Create the Dockerfile
FROM julia:latest
WORKDIR /env
COPY . . 
RUN julia -e 'using Pkg; Pkg.activate("."); Pkg.instantiate();'
  1. Remember to activate the environment on the container :slight_smile: 
(@v1.7) pkg> activate env
Julia> using HTTP
6 Likes