Documenter.jl doctest does not apply filter as expected

General Usage
, ,
1

Hi all,
I am writing the documentation for a package of mine, and in a doctest I have this output:

11-element Vector{ComplexF64}:
                     0.0 + 0.0im
   -1.617576049847552e-6 + 6.617439891157473e-23im
   -2.216802827443102e-6 - 2.602082008249062e-18im
  -2.1754485086580383e-5 + 5.293471330973909e-22im
    4.696594721193055e-5 + 6.511993073259912e-21im
   -6.763856924121541e-5 + 3.4410971976147374e-21im
   -3.686889866871368e-5 - 3.599741391814458e-21im
    2.176614750375809e-5 + 2.6487551060169705e-22im
   -6.419493464380101e-5 - 2.8434333556510326e-25im
 -0.00010698407746021099 + 6.775843525379591e-21im
    4.065995056214414e-5 + 6.776282561183226e-21im

a pretty standard array of complex numbers.
Now, I don’t care that much about the actual numbers, I just want to show a list of eleven complex numbers, so I assign the following substitution rule to the filter keyword of the doctest, :

r"^(\s+)?[-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)? [-+] [-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)?im"m => "### + ###im"

and set the expected output to

11-element Vector{ComplexF64}:
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im

As you can see, I have to deal not only with the actual floating-point numbers, but also to the variable white space at the beginning of each line.
That’s why I composed the regex with this structure in mind: “(variable amount of whitespace at beginning of line) (floating-point number) [-+] (floating-point number)im”. I put the m flag at the end of the raw string so that the ^ matches the start of each line instead of the start of the whole string only (it shouldn’t matter for the doctest, since, I think, each line is matched separately).

This works as expected in the REPL:

julia> str = """11-element Vector{ComplexF64}:
                            0.0 + 0.0im
          -1.617576049847552e-6 + 6.617439891157473e-23im
          -2.216802827443102e-6 - 2.602082008249062e-18im
         -2.1754485086580383e-5 + 5.293471330973909e-22im
           4.696594721193055e-5 + 6.511993073259912e-21im
          -6.763856924121541e-5 + 3.4410971976147374e-21im
          -3.686889866871368e-5 - 3.599741391814458e-21im
           2.176614750375809e-5 + 2.6487551060169705e-22im
          -6.419493464380101e-5 - 2.8434333556510326e-25im
        -0.00010698407746021099 + 6.775843525379591e-21im
           4.065995056214414e-5 + 6.776282561183226e-21im""";

julia> replace(str, r"^(\s+)?[-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)? [-+] [-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)?im"m => "### + ###im") |> print
11-element Vector{ComplexF64}:
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im
### + ###im

and yet the debug messages of Documenter show this:

Evaluated output (filtered):

11-element Vector{ComplexF64}:
                     0.0 + 0.0im
   -1.617576049847552e-6 + 6.617439891157473e-23im
   -2.216802827443102e-6 - 2.602082008249062e-18im
  -2.1754485086580383e-5 + 5.293471330973909e-22im
    4.696594721193055e-5 + 6.511993073259912e-21im
   -6.763856924121541e-5 + 3.4410971976147374e-21im
   -3.686889866871368e-5 - 3.599741391814458e-21im
    2.176614750375809e-5 + 2.6487551060169705e-22im
   -6.419493464380101e-5 - 2.8434333556510326e-25im
 -0.00010698407746021099 + 6.775843525379591e-21im
    4.065995056214414e-5 + 6.776282561183226e-21im

meaning that the regex filer didn’t work as expected.

I’m at a loss here, can some please help me and show me what I’m doing wrong?

2

I checked Documenter.jl source code, and apparently it only replaces the results IF both the expected output and the actual output match the regex: filter_doctests. Would you mind sharing both your jldoctest block and the output?

3

I managed to reproduce my problem just by drawing some random numbers, so that I don’t have to post the whole calculation that I had in my actual doctests.

This fails:

```jldoctest; filter = r"^(\s+)?[-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)? [-+] [-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)?im"m => "### + ###im"
julia> rand(ComplexF64, 7)                                                       
7-element Vector{ComplexF64}:                                                    
### + ###im                                                                      
### + ###im                                                                      
### + ###im                                                                      
### + ###im                                                                      
### + ###im                                                                      
### + ###im                                                                      
### + ###im                                                                      
```

Documenter outputs this debug message:

┌ Debug: Verifying doctest at docs/testrand.md
│ 
│ ```jldoctest; filter = r"^(\s+)?[-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)? [-+] [-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)?im"m => "### + ###im"
│ julia> rand(ComplexF64, 7)
│ 7-element Vector{ComplexF64}:
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ```
│ 
│ Subexpression:
│ 
│ rand(ComplexF64, 7)
│ 
│ Evaluated output:
│ 
│ 7-element Vector{ComplexF64}:
│   0.3666757812315634 + 0.5514356003090427im
│   0.8281494833007406 + 0.6011516785255695im
│   0.9634363838391149 + 0.24998991917806657im
│   0.7728861352300147 + 0.22854726370723033im
│  0.29978341620462245 + 0.5506865946612612im
│   0.5720278400709893 + 0.8265610770601598im
│   0.9441303898009322 + 0.4246940993558108im
│ 
│ Expected output:
│ 
│ 7-element Vector{ComplexF64}:
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ 
│ 1 doctest filter was applied:
│ 
│   r"^(\s+)?[-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)? [-+] [-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)?im"m => "### + ###im"
│ 
│ Evaluated output (filtered):
│ 
│ 7-element Vector{ComplexF64}:
│   0.3666757812315634 + 0.5514356003090427im
│   0.8281494833007406 + 0.6011516785255695im
│   0.9634363838391149 + 0.24998991917806657im
│   0.7728861352300147 + 0.22854726370723033im
│  0.29978341620462245 + 0.5506865946612612im
│   0.5720278400709893 + 0.8265610770601598im
│   0.9441303898009322 + 0.4246940993558108im
│ 
│ Expected output (filtered):
│ 
│ 7-element Vector{ComplexF64}:
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
│ ### + ###im
└ @ Documenter /opt/julia/packages/Documenter/AXNMp/src/doctests.jl:394

If I don’t pre-emptively filter the output myself, everything works fine: the following doctest compiles without errors, both expected and evaluated output are filtered producing the output with the #s.

```jldoctest; filter = r"^(\s+)?[-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)? [-+] [-+]?([0-9]*[.])?[0-9]+([eE][-+]?\d+)?im"m => "### + ###im"
julia> rand(ComplexF64, 7)           
7-element Vector{ComplexF64}:        
               0.0 + 0.0im
    -1.61757652e-6 + 6.6174473e-4im
    -2.21680202e-6 - 2.6020062e-5im
   -2.175448383e-5 + 5.2933909e-4im
     4.69659475e-5 + 6.5119912e-5im
    -6.76385641e-5 + 3.44107374e-6im
    -3.68688968e-5 - 3.5997458e-5im
```

(I realised then that this second doctest is actually what I want, because I don’t want to show the ###s in the documentation, but the actual numbers. Still, I expect Documenter not to throw an error in the first case…)

4

The reason that Documenter.jl throws an error is because filter is only meant to help with testing non-deterministic functions. In your first docstring, since the actual REPL outputs the numbers, but your expected output is ###s, it would error as expected.

The use case for filter is not for making documents aesthetically pleasing, for that you might as well consider using [...] to truncate output or use something else. It is used for things like:

julia> @time exp_func()
  0.03 seconds

Since the output of @time is non-deterministic, since I am testing the reliability of @time but not the speed of exp_func(), how many seconds it actually takes, in the expected output or in the evaluated output, doesn’t matter. Therefore filter = r"[0-9\.]+" => "***" could be used to mask (or remove) the parts i.e. the time that is not fixed across runs. So it would actually validate e.g. *** seconds against *** seconds instead of 0.03 seconds against 0.05 seconds.

Basically, filter is not intended for aesthetics but testing functions with unpredictable behaviour or special scenarios.

5

I see. So basically the filter regex has to match in both expected and evaluated cases. Thanks!