Different requirements for packages, using, building, and testing

General Usage
1

Is there a way of specifying distinct requirements for a package, for using it, for building, and for testing?

For example, I have a package that needs LightXML, but only for running deps/build.jl, which downloads one or more data files, stores them in a data subdirectory, and then produces a data file which it also stores there, which is used by the package. The package itself does not need LightXML, and I don’t want the extra overhead of that dependency just to use the package. The same is true for testing, where we want BaseTestNext (for v0.4.7) and BenchmarkTools, but only when running the unit tests, not for deployment.

If there is no way of handling those issues, will that be something addressed by the Pkg3 redesign?
Also, is it acceptable for the build.jl or runtests.jl to check to see if the packages(s) are already present, and if not,
call Pkg.add (and possibly Pkg.build), before using the package(s), in order to keep those out of REQUIRES?

1 Like
2

You can have REQUIRE in test additionally.
Packages should - as far as i understood some discussions in the METADATA PRs - never run Pkg.add on their own.

And i agree with you, all this can take you to dependency hell.

3 Likes
3

Thanks! That will help some.

You can’t have a separate REQUIRE in deps? That’s the one that causes the most problems.

4

Could you explain, what problem you actually try to solve?
The deps/ dir is part of the BinDeps setup, so you should find help for installation specifics there first. Then i don’t see LightXML as heavy dependency for building a package, but if you disagree why not commiting the data file directly to the package?
And i have some problems to understand the logic: To run/use a package it needs to be build locally, so in this case LightXML is a valid, required dependency. To have additional requirements for testing, makes some sense, as not everyone needs to run testing on installed packages, but everyone runs a build.

5

I’m not using BinDeps at all (that’s for binary dependecies AFAIK, it just hooks into the general deps/build.jl setup), deps/build.jl is called by Pkg.build.

LightXML (for one of the packages) and JSON (for some of the others) are only needed for building the data tables, after that point, they are not needed at all to use the package.
The data tables only need to be rebuilt when something changes, like a new release of Unicode, say when 10.0 comes out, or if somebody wants to manually add an entity to the LaTeX entity tables. (I need to add that capability to manually add entities to the other Emoji and HTML entity tables as well).

I suppose I can check in the current state of the data tables, and rename deps/build.jl to something else, so that it won’t get called by Pkg.build() automatically, and document how it can be called to rebuild the data table.

6

Do you assume that it’s possible to use/run a package without building it?

7

The build script downloads the input data files (for Unicode, HTML, Emojis, or LaTeX entities respectively, for each package) if they are not already present, and processes them to produce the data file that is needed when the package is used/run. None of that is needed for deploying the package, just the final output data file.

In the emoji_symbols.jl and latex_symbols.jl files, the scripts used to create them (which are out of data and no longer work) were included as comments in the output files (instead of producing compact data files, they produce julia code that needs to be compiled and run, which adds a lot of extra time / unreclaimable space (since julia code, AFAIK, is not GCed).

What I’m trying to do is make the cost of having loadable tables (which also allow one to update them between releases) much smaller, by separating out the process of building the tables and creating the Julia structures, without requiring unnecessary run-time dependencies.

8

Isn’t this what got discussed in: Development, Test, and Production Environments?

with it you can do something like this for REQUIRE:

gem 'httparty'

group :development, :test do
  gem 'better_errors'
end

group :production do
  gem 'postgres'
end

// edit: i just used a gemfile instead of a require file because i had one on hand

9

I have a similar issue that I would like to solve. The package that I’m developing does not use the dependency InfoZip. However the tests require downloading some data and I require using InfoZip to run the tests.

I have a similar need in another where Pkg.build adds additional data optionally, that is not required for the main package to run. Is there a way for specifying build requirements, test requirements and package run time requirements separately?

10

Julia 0.6:
https://docs.julialang.org/en/stable/manual/packages/#REQUIRE-speaks-for-itself-1

Julia 0.7:
https://docs.julialang.org/en/latest/stdlib/Pkg/#Test-specific-dependencies-1

11

Thanks! That was helpful! I will try that out.

12

@fredrikekre What about build dependencies? If I have a requirement for Pkg.build that’s not part of the main package what should I do. I essentially want to download and unzip a large zip file with a lot of data before running any tests.

Pkg.clone("https://github.com/Org/Package")
Pkg.build("Package") # fails because dependency wasn't installed
Pkg.test("Package") # Installs dependency

Suggestions?

13

Pkg.build is run by everyone installing the package – it needs to be run before the package can be used.
(unlike Pkg.test).
thus anything you need for building, is simply something you need.

So it goes in the main dependencies.

14

That’s not entirely true: build deps need to be present at build time but don’t need to remain installed and even more importantly don’t need to be loaded into the same Julia process so they don’t need to be compatible with all of a project’s dependencies. You can add a build target the same way you can add a test target.

1 Like
15

That is cool!

Can I have arbitrary targets, keyed to any command run by Pkg?

Could I have say a dev target that is for dependencies needed during development.
For example static code generation, the output of which is checked into version control.

Or is it just build and test?

16

You can have any kind of target: they’re named subsets of deps + extras (which must include deps). The test and build targets happen have special meaning to the package manager. In the future we might have other ones that it knows about but the mechanism for expressing them is the same.