Proper way to create sysimage with all dependencies of a given package

General Usage
,
1

What’s the proper way to create a custom sysimage for the following situation:

I am developing a package with at least a moderately large set of dependencies (Trixi.jl) and I would like to speed up startup latency by compiling all dependencies into a sysimage with PackageCompiler.jl. How do I properly invoke PC.jl such that all dependencies are included but not my package itself (since I am actively developing it)?

Right now, I am doing something like the following, where I copy the Project.toml from my package to another folder, remove the everything but the [deps] and [compat] sections, instantiate it, then install my package in the same environment, and finallly run PC.jl with a script that exercises some functions of my package that in turn make use of the dependencies.

This seems like overly convoluted, thus I am sure that there must be an easier way to do it, right? During development, I usually run julia --project in my cloned package directory, and ideally I could just do that with a “special” PC.jl invocation to create a suitable sysimage.

3 Likes
2

You can specify the packages for which the sysimage is built as the first argument to create_sysimage

create_sysimage(packages::Vector{String}; kwargs...)

Create a system image that includes the package(s) in packages (given as a string or vector). If the packages argument is not passed, all packages in the project will be put into the sysimage.

3

Thanks for the reply! Yes, I am aware of this argument, and this is what I currently use as well. However, I am wondering if there really is no other solution than to extract the package names manually from the Project.toml. I am looking for something (logically, not literally) like create_sysimages(depsof(Trixi), ...) where depsof will return all dependencies of a given package, or alternatively where I can include all packages in the current project except a certain package.

4

You can use Pkg.dependencies() to get a dictionary of all packages in the environment, and massage that into a vector of names.

depsof(package::String) = filter(!=(package), map(p->p.name, values(Pkg.dependencies())))
5

Is there a way to really get only the direct dependencies of a given package? For example, sometimes I might have other, auxiliary packages in the same environment, such as OrdinaryDiffEq.jl.

6

You can do something like:

julia> uuid_dataframes = only(findall(x -> x.name == "DataFrames", Pkg.dependencies()))
UUID("a93c6f00-e57d-5684-b7b6-d8193f3e46c0")

julia> Pkg.dependencies()[uuid_dataframes].dependencies
Dict{String, Base.UUID} with 18 entries:
  "Statistics"                  => UUID("10745b16-79ce-11e8-11f9-7d13ad32a3b2")
  "SortingAlgorithms"           => UUID("a2af1166-a08f-5f64-846c-94a0d3cef48c")
  "PooledArrays"                => UUID("2dfb63ee-cc39-5dd5-95bd-886bf059d720")
  "TableTraits"                 => UUID("3783bdb8-4a98-5b6b-af9a-565f29a5fe9c")
  "REPL"                        => UUID("3fa0cd96-eef1-5676-8a61-b3b8758bbffb")
  "InvertedIndices"             => UUID("41ab1584-1d38-5bbf-9106-f11c6c58b48f")
  "LinearAlgebra"               => UUID("37e2e46d-f89d-539d-b4ee-838fcccc9c8e")
  "Missings"                    => UUID("e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28")
  "IteratorInterfaceExtensions" => UUID("82899510-4779-5014-852e-03e436cf321d")
  "Compat"                      => UUID("34da2185-b29b-5c13-b0c7-acf172513d20")
  "Tables"                      => UUID("bd369af6-aec1-5ad0-b16a-f7cc5008161c")
  "Unicode"                     => UUID("4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5")
  "DataAPI"                     => UUID("9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a")
  "Future"                      => UUID("9fa8497b-333b-5362-9e8d-4d0656e87820")
  "Printf"                      => UUID("de0858da-6303-5e67-8744-51eddeeeb8d7")
  "Markdown"                    => UUID("d6f4376e-aef5-505a-96c1-9c027394607a")
  "PrettyTables"                => UUID("08abe8d2-0d0c-5749-adfa-8a2ac140af0d")
  "Reexport"                    => UUID("189a3867-3050-52da-a836-e630ba90ab69")
4 Likes
7

In a related project, we’ve used the following to automatically extract and filter the dependencies from an existing environment:

	# Get all packages from the current project
	all_packages = String[]
	for (uuid, dep) in Pkg.dependencies()
		dep.is_direct_dep || continue
		dep.version === nothing && continue
		push!(all_packages,dep.name)
	end

	# Remove unneeded packages
	do_not_include = ["PackageCompiler","SIEGE"]
	package_list = filter(x -> x ∉ do_not_include, all_packages)
2 Likes
8

What, if the UUID is not unique. This the case if my local clone of a project should be taken instead of the version that is public and has the same UUID?

9

This method would just use whichever version is in the current environment. I don’t think it’s possible to have both. For instance, when I try the following, adding the local version just replaces the public version, and vice versa.

pkg> activate --temp
pkg> add MyPackage # public version
pkg> dev /path/to/local/MyPackage
10

This is not sufficient :frowning: See: Missing Relocatability of Tulip - #9 by ellocco