Different requirements for packages, using, building, and testing



Is there a way of specifying distinct requirements for a package, for using it, for building, and for testing?

For example, I have a package that needs LightXML, but only for running deps/build.jl, which downloads one or more data files, stores them in a data subdirectory, and then produces a data file which it also stores there, which is used by the package. The package itself does not need LightXML, and I don’t want the extra overhead of that dependency just to use the package. The same is true for testing, where we want BaseTestNext (for v0.4.7) and BenchmarkTools, but only when running the unit tests, not for deployment.

If there is no way of handling those issues, will that be something addressed by the Pkg3 redesign?
Also, is it acceptable for the build.jl or runtests.jl to check to see if the packages(s) are already present, and if not,
call Pkg.add (and possibly Pkg.build), before using the package(s), in order to keep those out of REQUIRES?

Separate REQUIRE for deps/build.jl

You can have REQUIRE in test additionally.
Packages should - as far as i understood some discussions in the METADATA PRs - never run Pkg.add on their own.

And i agree with you, all this can take you to dependency hell.


Thanks! That will help some.

You can’t have a separate REQUIRE in deps? That’s the one that causes the most problems.


Could you explain, what problem you actually try to solve?
The deps/ dir is part of the BinDeps setup, so you should find help for installation specifics there first. Then i don’t see LightXML as heavy dependency for building a package, but if you disagree why not commiting the data file directly to the package?
And i have some problems to understand the logic: To run/use a package it needs to be build locally, so in this case LightXML is a valid, required dependency. To have additional requirements for testing, makes some sense, as not everyone needs to run testing on installed packages, but everyone runs a build.


I’m not using BinDeps at all (that’s for binary dependecies AFAIK, it just hooks into the general deps/build.jl setup), deps/build.jl is called by Pkg.build.

LightXML (for one of the packages) and JSON (for some of the others) are only needed for building the data tables, after that point, they are not needed at all to use the package.
The data tables only need to be rebuilt when something changes, like a new release of Unicode, say when 10.0 comes out, or if somebody wants to manually add an entity to the LaTeX entity tables. (I need to add that capability to manually add entities to the other Emoji and HTML entity tables as well).

I suppose I can check in the current state of the data tables, and rename deps/build.jl to something else, so that it won’t get called by Pkg.build() automatically, and document how it can be called to rebuild the data table.


Do you assume that it’s possible to use/run a package without building it?


The build script downloads the input data files (for Unicode, HTML, Emojis, or LaTeX entities respectively, for each package) if they are not already present, and processes them to produce the data file that is needed when the package is used/run. None of that is needed for deploying the package, just the final output data file.

In the emoji_symbols.jl and latex_symbols.jl files, the scripts used to create them (which are out of data and no longer work) were included as comments in the output files (instead of producing compact data files, they produce julia code that needs to be compiled and run, which adds a lot of extra time / unreclaimable space (since julia code, AFAIK, is not GCed).

What I’m trying to do is make the cost of having loadable tables (which also allow one to update them between releases) much smaller, by separating out the process of building the tables and creating the Julia structures, without requiring unnecessary run-time dependencies.


Isn’t this what got discussed in: Development, Test, and Production Environments?

with it you can do something like this for REQUIRE:

gem 'httparty'

group :development, :test do
  gem 'better_errors'

group :production do
  gem 'postgres'

// edit: i just used a gemfile instead of a require file because i had one on hand