PSA: Use a project for building your docs

This is a simple guide on how to best build your documentation in Julia v1.0 if you are using Documenter.

The old way

If you are using Documenter you probably have something like the following in your .travis.yml file for building your documentation:

after_success:
  - julia -e 'Pkg.add("Documenter")'
  - julia -e 'cd(Pkg.dir("PACKAGE_NAME")); include(joinpath("docs", "make.jl"))'

Why is this bad? Here are some reasons:

  • There is no good way to add doc-build dependencies, you have to add them manually (like we Pkg.add("Documenter") above).
  • Code in the after_success: part of the build does not fail the build. This makes it difficult to verify that (i) the doc build works and (ii) that doctest etc. still passes. For this reason some package have chosen to build the docs as part of the test suite instead.
  • Doc building runs in the global environment, and is thus affected by the surroundings.

So, maybe we can use this new Pkg thing, that apparently should solve all our problems. Absolutely!

The new way

In Julia v1.0 we can instead use a designated doc-build project with a Project.toml in the docs/ directory. The project includes Documenter and all your other doc-build dependencies. If Documenter is the only dependency, then the Project.toml file should include the following:

[deps]
Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"

[compat]
Documenter = "~0.19"

Here we also added a compatibility constraint for Documenter, since I happen to know that Documenter 0.20 will break everything.

Instead of creating the Project.toml file manually we could have started Julia with julia --project=docs/ (alternatively just start Julia and then using Pkg; Pkg.activate("docs")) and then used the Pkg REPL to add our doc-build dependencies.

Next we need to configure the .travis.yml to use this project. For this we will utilize Travis Build Stages that have just been released (see Build Stages - Travis CI). Add the following to your .travis.yml file:

jobs:
  include:
    - stage: "Documentation"
      julia: 1.0
      os: linux
      script:
        - julia --project=docs/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd()));
                                               Pkg.instantiate()'
        - julia --project=docs/ docs/make.jl
      after_success: skip

This adds a new build stage, called Documentation, that will run after the default test stage. It will look something like this. The julia: and os: entries control from which worker we build the docs, for example Julia 1.0 and linux. What happens on the three lines in the script: part?

  1. The first line Pkg.instantiate()s the project, meaning that Documenter and the other dependencies will be installed.
  2. The second line adds our package to the doc-build environment.
  3. The third line builds the docs.

Lastly, commit the files (the Manifest.toml can be .gitignored) and push!

Why is this better?

  • Using a custom docs projects gives full control over the environment where the docs build.
  • We can have documentation dependencies.
  • A failed doc build now fails CI.

In my opinion this is a pretty nice example of how to utilize the new Pkg environments.

If you need some inspiration, have a look at the following packages that already uses projects and build stages for doc-building:

Some gotchas

  1. Make sure that the julia and os arguments to Documenter.deploydocs match the configuration you define for the build stage in .travis.yml. Note that this is something that will change in the next Documenter release, where the julia and os arguments are removed and the deployment is essentially governed by the .travis.yml configuration. So the annoyance of keeping those in sync is temporary. EDIT: This does not apply to Documenter v0.20.

  2. If you are using doctests, you will run into absolutify --project path by fredrikekre · Pull Request #28625 · JuliaLang/julia · GitHub (you will see something like ERROR: LoadError: ArgumentError: Package Foo not found in current path) which is a bug in Julia v0.7, v1.0 (but fixed on nightly and the upcoming Julia v1.0.1). You can add the following lines at the top of docs/make.jl to workaround it:

    # Workaround for JuliaLang/julia/pull/28625
    if Base.HOME_PROJECT[] !== nothing
        Base.HOME_PROJECT[] = abspath(Base.HOME_PROJECT[])
    end
    

Feel free to ping me (@fredrikekre) for any questions from your PRs. Happy upgrading!

40 Likes

Awesome, thanks!

Could this be integrated into the default travis build script, so that it just automatically runs if there is a docs/make.jl file?

1 Like

I am pretty sure the Julia Travis script can’t create a new build stage. Also, you need need to provide explicit Julia version and OS information to the build stage.

since I happen to know that Documenter 0.20 will break everything.

This is persuading me to leave everything as it is for now, and address all the breakages and upgrades at the same time…

2 Likes

Just to clarify: The next Documenter release will not break the setup I am suggesting here. It will however require some changes to the docs/make.jl file. Actually, adding the compatibility constraint like I am suggesting will protect you from the breakage, since if you blindly Pkg.add("Documenter") you will get the breaking version when that is released.

3 Likes

It should be noted that if Manifest.toml is checked in then the exact version of those packages will be used every time the documentation is built. This means that generating the documentation is reproducible and isn’t at risk of breaking just because someone released a new (perhaps broken) version of one of your documentation dependencies.

2 Likes

Right, maybe that is actually a better option but then you have to explicitly upgrade your doc build dependencies to get shiny new stuff. I actually do this in Literate.jl, where I have developed the package with a relative path, stored in the manifest. That means that Pkg.instantiate() is enough, and I don’t have to Pkg.add my own package as a separate step.

Thanks @fredrikekre! Makes doc building more reliable and reproducible.

Unfortunately it means that the package has to be built one more time, which was not the case with the old approach. Do you know if there is a way around? It feels a bit like a waste of CI resources, since it was built already.

Right, so this obviously works best for packages that do not have a complicated build setup. For simpler packages the approach I proposed here have a very small overhead.

You can either define a new job, which then only build the documentation, and will run in parallell with the other jobs from your matrix expansion. This will obviously also build the package again, but at least in parallell with the other jobs.

The other alternative is to just append the doc building to the default script: part. Be careful to filter out the correct os and julia version though. Something like

- if [ ${TRAVIS_OS} == "linux"] && [ ${TRAVIS_JULIA_VERSION} == "1.0" ]; then
    julia --project=docs/ -e 'using Pkg; Pkg.instantiate(); Pkg.add(PackageSpec(path=pwd()))'
    julia --project=docs/ docs/make.jl
  fi

might work, but I have not tested this.

1 Like

I’m trying to migrate the style of documentation but it failed due to installation failure. My pakcage requires Pkg.build to install a binary library via PackageProvider.jl; however, running Pkg.build(pkgspec) always results in the following error:

 Resolving package versions...
 Installed BinaryProvider ─ v0.5.2
  Updating `~/build/bicycle1885/EzXML.jl/docs/Project.toml`
  [8f5d6c58] + EzXML v0.9.0+ [`~/build/bicycle1885/EzXML.jl`]
  Updating `~/build/bicycle1885/EzXML.jl/docs/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.2
  [8f5d6c58] + EzXML v0.9.0+ [`~/build/bicycle1885/EzXML.jl`]
ERROR: LoadError: The following package names could not be resolved:
 *  (not found in project or manifest)
Please specify by known `name=uuid`.
Stacktrace:
 [1] pkgerror(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:120
 [2] #ensure_resolved#43(::Bool, ::Function, ::Pkg.Types.EnvCache, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:896
 [3] ensure_resolved at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:866 [inlined]
 [4] #build#53(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:447
 [5] build at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:428 [inlined]
 [6] build(::Pkg.Types.PackageSpec) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:425
 [7] top-level scope at none:0
 [8] include at ./boot.jl:317 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1041
 [10] include(::Module, ::String) at ./sysimg.jl:29
 [11] exec_options(::Base.JLOptions) at ./client.jl:229
 [12] _start() at ./client.jl:421
in expression starting at /home/travis/build/bicycle1885/EzXML.jl/docs/setup.jl:7

Complete job log: Travis CI - Test and Deploy Your Code with Confidence.

Here is the pull request where I’m trying to build the docs: https://github.com/bicycle1885/EzXML.jl/pull/97.
The install script is docs/setup.jl (a few @shows are for debugging).

I have no clue to debug this problem, so any advice would be appreciated.

The problem is the Pkg.build call; turns out you can not give a PackageSpec which only specifes a path to Pkg.build. You can either (i) comment out the build call since Pkg should build the package in dev or (ii) change it to Pkg.build("EzXML").

Thank you. I’ve tried the option 1 but it failed due to “EzXML.jl is not installed properly” error (Travis CI - Test and Deploy Your Code with Confidence), and that’s why I added Pkg.build(pkgspec) to the last line. If this is not an intended behavior, perhaps this may be a bug of Pkg or something.

EDIT: I filed an issue at Pkg.jl: https://github.com/JuliaLang/Pkg.jl/issues/862.

Go with Pkg.build("EzXML") then.

2 Likes

Travis CI is now happy with Pkg.build("EzXML"). Thank you very much :wink:

1 Like

We have noticed that Travis passes even when doctests fails on JuMP.
Any idea what might be the issue ?

You need strict = true for Documenter to fail the build (the doc build passes when there are failing doctests · Issue #1576 · jump-dev/JuMP.jl · GitHub).

Question: if I have the root Project.toml contains the dependency for a package A that I need for the docs to compile, but the docs/Project.toml does not, then I should expect using A to fail while building the docs, is this correct?

Is there a way to “merge” the root Project.toml without duplicating the information in it?

Yes, you will only be able to load stuff from docs/Project.toml (and also environments/v1 and @stdlib since those are also in the default load path).

You can put the root Project.toml file in the load path after docs/Project.toml. From the root of the package:

export JULIA_LOAD_PATH=docs/:.:
julia -e 'using Pkg; Pkg.instantiate()'
julia docs/make.jl

However, I would argue that it is better to just duplicate the deps for docs/Project.toml, you only install what you actually need in the doc build environment. Also, when writing the docs you should interact with the package from “the outside” and only use package that the user also has, if that makes sense.

Thanks. It is this package:

The “package” is just used to provide a relative path. Then some examples are run and the results are checked, and also the same scripts are passed through Literate.jl for a HTML output.

Hope it is OK to necro the topic with a question: if a package uses doctests for (some of the) coverage, what is the recommended setup?

My understanding is that with the setup posted above, the docstests will not contribute to coverage.

Perhaps I am missing something, an example to a package that does this would be useful.