Documenter/CI

New to Julia
1

Hi,

First, let me say that Julia’s packaging (Pkg, PkgTemplates, etc…) is very impressive but also too difficult for dumb mortals like me to understand, unfortunately. I’ve got a package I’m trying to make conform a bit more to standard Julia things. I’ve already got docstrings everywhere. I’m trying to add Documenter to this. I thought I did it right, but now I’m getting error messages that I can’t fathom.

There is so much state to this thing (Documenter specifically, and all the configuration files for my project) that I have no idea what I should post here so that someone can help me. I’ve decided to push my broken thing to github and put the github link: GitHub - sloisel/MultiGridBarrier.jl: MultiGrid Barrier method

I’ll stick around for a little while and read replies, but I’m going to bed soon. I’m secretly hoping I did something really dumb and easy and someone will have posted the solution when I next check, but in case I only respond after I sleep, apologies.

Now for the error messages:

sebastienloisel@macbook-pro docs % julia --project make.jl       

[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
[ Info: ExpandTemplates: expanding markdown templates.
┌ Error: duplicate docs found for 'MultiGridBarrier.AMG' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.Barrier' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.amg' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.amgb' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.barrier' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.damped_newton' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.fem1d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.fem2d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.fem_interp1d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.fem_interp1d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.fem_plot_2d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.fem_solve_1d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.fem_solve_2d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.interp1d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.interp2d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.plot1d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.plot2d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.spectral_solve_1d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
┌ Error: duplicate docs found for 'MultiGridBarrier.spectral_solve_2d' in src/index.md:12-14
│ ```@autodocs
│ Modules = [MultiGridBarrier]
│ ```
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/utilities/utilities.jl:44
[ Info: CrossReferences: building cross-references.
[ Info: CheckDocument: running document checks.
[ Info: Populate: populating indices.
ERROR: LoadError: `makedocs` encountered an error [:autodocs_block] -- terminating build before rendering.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] runner(::Type{Documenter.Builder.RenderDocument}, doc::Documenter.Document)
   @ Documenter ~/.julia/packages/Documenter/CJeWX/src/builder_pipeline.jl:253
 [3] dispatch(::Type{Documenter.Builder.DocumentPipeline}, x::Documenter.Document)
   @ Documenter.Selectors ~/.julia/packages/Documenter/CJeWX/src/utilities/Selectors.jl:170
 [4] #86
   @ ~/.julia/packages/Documenter/CJeWX/src/makedocs.jl:248 [inlined]
 [5] withenv(::Documenter.var"#86#88"{Documenter.Document}, ::Pair{String, Nothing}, ::Vararg{Pair{String, Nothing}})
   @ Base ./env.jl:257
 [6] #85
   @ ~/.julia/packages/Documenter/CJeWX/src/makedocs.jl:247 [inlined]
 [7] cd(f::Documenter.var"#85#87"{Documenter.Document}, dir::String)
   @ Base.Filesystem ./file.jl:112
 [8] #makedocs#84
   @ ~/.julia/packages/Documenter/CJeWX/src/makedocs.jl:247 [inlined]
 [9] top-level scope
   @ ~/Dropbox/2023/Spectral Barrier Method/julia/MultiGridBarrier.jl/docs/make.jl:6
in expression starting at /Users/sebastienloisel/Dropbox/2023/Spectral Barrier Method/julia/MultiGridBarrier.jl/docs/make.jl:6
sebastienloisel@macbook-pro docs %
2

I’m not sure why you get that error message locally. Usually it means that you have included the same docstring twice, but I don’t see duplicates in your index.md.
In any case, the Documentation CI run fails even earlier, when you try to install PyPlot.jl. If you want to avoid the struggle with Python dependencies, may I suggest switching to a pure-Julia solution like Makie.jl, or something that is easier to install like Plots.jl? The syntax is very similar.

As for the packaging tools in general, it is true that they are a bit daunting for beginners. Some friends and I wrote a blog to try and make sense of it all, hopefully it can help:

https://modernjuliaworkflows.github.io/sharing/

3

Thanks for your reply.

I know that CI on github fails in an altogether different way, I wasn’t asking about that yet but your information is valuable. This code goes with a couple of papers and the choice of PyPlot was made after trying several plotting packages, this was the only suitable choice at the time. Unfortunately, if PyPlot is incompatible with CI, I will have to forgo CI. :frowning:

4

As you can see here: GitHub - aenarete/ControlPlots.jl: Create plots that can be saved and edited with Julia PyPlot works fine with CI …

1 Like
5

For the PyPlot error, at first glance it seems to be missing from the environment in docs/ where that should be present – and also loaded in the make.jl.

For the duplicate docs I would have to clone the repo, can try to do that later today and take a look, but at first glance I am surprised those errors to appear (and no worries, that is not a dumb question to ask for help with error messages!).

1 Like
6

I cloned your repository. Had to change the first few lines of the docs to

using Pkg
Pkg.activate(@__DIR__)
# As long as it is not registered, this is nice, in general it locally always
# renders docs of the current version checked out in this repo.
Pkg.develop(PackageSpec(; path=(@__DIR__) * "/../"))
using MultiGridBarrier
using Documenter, PyPlot

Since your package is not registered and this way documenter always takes the parent folder as the dev version of your package, so that way documenter always renders the docs belonging to the current state of the repo.

Without those lines I get the error that your package is unknown (sure, it is not registered). I also added PyPlot to the docs env.

My “problem” now is, that on my machine your docs just render fine

julia> include("make.jl")
  Activating project at `~/Repositories/Julia/MultiGridBarrier.jl/docs`
   Resolving package versions...
  No Changes to `~/Repositories/Julia/MultiGridBarrier.jl/docs/Project.toml`
  No Changes to `~/Repositories/Julia/MultiGridBarrier.jl/docs/Manifest.toml`
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
[ Info: ExpandTemplates: expanding markdown templates.
[ Info: CrossReferences: building cross-references.
[ Info: CheckDocument: running document checks.
[ Info: Populate: populating indices.
[ Info: RenderDocument: rendering document.
[ Info: HTMLWriter: rendering HTML pages.
[ Info: Automatic `version="1.0.0-DEV"` for inventory from ../Project.toml
┌ Warning: Documenter could not auto-detect the building environment. Skipping deployment.
└ @ Documenter ~/.julia/packages/Documenter/CJeWX/src/deployconfig.jl:76

which is what I expected from them. So could you provide a bit more details how you render the docs locally and which version of Documenter you are using?

For the above run, my environment in docs looks like

(docs) pkg> status
Status `~/Repositories/Julia/MultiGridBarrier.jl/docs/Project.toml`
  [e30172f5] Documenter v1.4.1
  [9e2c1f1d] MultiGridBarrier v1.0.0-DEV `~/Repositories/Julia/MultiGridBarrier.jl`
  [d330b81b] PyPlot v2.11.2

and I either run the docs like you with julia --project make.jl (in the docs folder) or include("make.jl") (after activating the docs env manually). Both yield the output above.

7

Hi,

Thanks for spending time on my problem like this.

I’ve changed the first few lines of the docs to what you wrote, and also added PyPlot to the docs environment like you did, then reran include("make.jl") as you suggest, but I got the same error message as before.

First question: why do I have to add PyPlot to my environment, and also using it in make.jl? And why don’t I have to do that with other packages that are used by MultiGridBarrier, for example, ProgressMeter, QuadratureRules, ForwardDiff, SparseArrays, etc…?

I ran status as you requested, here it is:

(docs) pkg> status
Status `~/Dropbox/2023/Spectral Barrier Method/julia/MultiGridBarrier.jl/docs/Project.toml`
  [e30172f5] Documenter v1.4.1
  [9e2c1f1d] MultiGridBarrier v1.0.0-DEV `~/Dropbox/2023/Spectral Barrier Method/julia/MultiGridBarrier.jl`
  [d330b81b] PyPlot v2.11.2
8

Ehm, the `PyPlot was mainly in preparation because I thought that would fix the other error later on CI as well, but I am not sure about that yet. So that is maybe one step too far ahead trying to solve the other CI problem as well.

Hm, since the status the same, there must be something different in your docs/src folder that we haven’t found yet. Since it is working on my side (with the version on GitHub) – are you sure your local repository is exactly the same as what is on GitHub?

(Btw I would not recommend to put a git repo in a Dropbox; dropbox sync can have troubles with the .git folder, yielding a sync conflict could invalidate your local repo)

edit: Or to make the question more precise: Is there any other .md file locally present in your docs/src? All files present there will be rendered to HTML (processed by Documenter) – independent on whether they are listed in the menu or not. So if you have a list of functions in a a_test.md that will cause your errors.

9

Thanks again for your help.

From my previous experiences with Julia, and from your description that you could, in fact, build my documentation locally on your machine, I suspected that my Julia installation’s state got borked again. For background, in my ordinary use of Julia, I often find myself with a broken installation. For example, if I accidentally ]add MultiGridBarrier.jl instead of ]dev, I found it really hard to recover from this.

So I nuked everything and started from scratch. I did brew uninstall julia and rm -rf ~/.julia and then also rm -rf MultiGridBarrier.jl. I reinstalled everything. Now I can build my documentation locally.

So I don’t know exactly what it was, but something in the Julia state on my machine was broken.

Now I progress to the next thing that is broken for me: CI on github, for lack of a better name? Right now, github is failing to build docs. How do I get it to build docs?

This is what I’m seeing: Merge pull request #2 from sloisel/dependabot/github_actions/julia-ac… · sloisel/MultiGridBarrier.jl@f8c9cc6 · GitHub

The error message is:

ERROR: LoadError: InitError: PyError (PyImport_ImportModule
The Python package matplotlib could not be imported by pyimport. Usually this means

What’s the fix for this?

Thanks,

S

10

]add MultiGridBarrier.jl should error, so should ]add MultiGridBarrier (both are not registered packages, if it would be registered, the second is the right way to add it); in both ]rm MultiGridBarrier resolves the issue.

But yeah you should be able to run locally (for me your first post even read like you did run it locally)

Yeah, it seems that maybe your dependencies are not loaded correctly (especially something related to Python). For that I do not see directly the solution – I could help with the Documenter issue, but with Python? I am fighting quite a bit as well (and try to avoid it if possible), so I am sorry I can not provide much help on that.

11

Thanks again for all your help. I was finally able to get the documentation to build on GitHub, and also for the CI tests to run. For posterity, I am including the final steps of the solution.

So @kellertuer is the one who identified that my Julia installation was broken, so it was essential that I uninstall Julia and also rm -rf ~/.julia. I also nuked my local MultiGridBarrier.jl repository and re-cloned it from GitHub, for good measure.

It was @ufechner7 who gave the helpful suggestion to look at the following repository: GitHub - aenarete/ControlPlots.jl: Create plots that can be saved and edited with Julia

In that repository’s .github/workflows/CI.yml, I discovered the following fragment:

...
    steps:
      - name: Install matplotlib
        run: if [ "$RUNNER_OS" = "Linux" ]; then sudo apt-get install -y python3-matplotlib; fi
        shell: bash
...

I added this to my CI.yml twice, once for the test “target”, and once for the docs “target”.

I’m not exactly sure what is so special about the GitHub environment that MatPlotLib, and only MatPlotLib, has to manually be installed.

Also, thank you to @kellertuer for your suggestion to rm my package, I had tried this and it had left my Julia installation in an inconsistent state. There were inconsistencies at minimum between the following files and folders, in ~/.julia: Manifest.toml and Project.toml, and the packages directory all contained problematic data, after I had done rm MultiGridBarrier. I mention this for posterity, in case someone lands on this discussion through a Google search in the future, wondering why their Julia installation isn’t working after they accidentally added and rmed a package they intended to dev.

Edit: also, @kellertuer suggested to add something to docs/make.jl, at the top, and I did this, and it may be important to do so, it does make sense to me:

using Pkg
Pkg.activate(@__DIR__)
# As long as it is not registered, this is nice, in general it locally always
# renders docs of the current version checked out in this repo.
Pkg.develop(PackageSpec(; path=(@__DIR__) * "/../"))

There was no need to add PyPlot anywhere in the docs/ subdirectory.

12

Great, that you solved it!
As a remark, inconsistencies between the project and the manifest are often resolved … with ] resolve.

13

Thanks for your suggestion. I had tried ]resolve, and it does resolve some problems between project.toml and manifest.toml (although maybe sometimes not all problems?), but it did not resolve problems with the packages folder.

14

OK one more for posterity.

Documenter looks recursively through the docs/src subdirectory for documentation to generate. Jupyter Lab created a hidden subdirectory docs/src/.ipynb_checkpoints, and I guess this subdirectory contained a backup of docs/src/index.md, and hence Documenter believed that all documentation had been emitted twice.

Long story short, Jupyter is incompatible with Documenter.

15

Oh! But that clarifies the duplicates that you had but I did not.

As I wrote above – yes, Documenter renders all .md files in the source folder.
I am not sure how Jupyter came to the idea to duplicate that, but that then was the origin of the errors this thread started with.

What did you use Jupyter for in the docs? I always write my Markdown in an editor directly.

16

I just use Jupyter Lab. It’s not great but I had trouble with plots in VS code.

17

With respect to “what makes the github environment special”, the OS image you are probably running on CI has a default python installation. The apt-get command installs a version of matplotlib from ubuntu’s repositories into this default python installation.

You can have more control over this process (e.g., Python and matplotlib versions). I used the following steps on a project I set up recently. These steps install the most recent minor version of python 3.x, update pip, and use pip to install the latest patch version of matplotlib 3.8. There is currently a bug with matplotlib 3.9 in PyPlot.jl which there is an open PR to fix. As Github runners have many python versions cached, the setup process is still very quick.

      - uses: actions/setup-python@v5
        with:
          python-version: '3.x'
      - run: python -m pip install --upgrade pip
      - run: pip install 'matplotlib==3.8.*'

I run this directly after checking out the repository, before building PyPlot.jl, so that it will find and use this new version instead of the OS default.

2 Likes
18

I’ve made this change. I don’t know if it helps anything, but it seemed less quirky than the apt-get I had. Thanks.