The challenges of documenting generic functions

General Usage
1

First of all, I must apologize, in advance, for the blunt language below. I’m not going to accuse anybody. I’ll just try to describe the problems I’ve encountered.

I have been using Julia for 1.5 years (and I’m sorry to skip great things about Julia) and the greatest difficulty in using Julia is the general “lack” of “manuals”.

Tutorials are excellent. But, to find out details of the arguments to library functions is sometimes extremely hard.

Most of julia’s core standard functions are very general and as a result, the most “accurate” description of the API must needs be abstract. For example, Base.findall is described (defined) as

Return a vector I of the true indices or keys of A

There is no clue (to a newcomer) what A is expected to be. The examples given there all use a vector for A. From some of the examples given there, I got the (incorrect) impression that the A must be an array or vector. After a 1.5-year experience with Julia, I can tell that ranges and many more collections can works as A.

But, when I was a complete newbie, I naïvely thought that a vector is expected and wrote findall(somecondition, collect(1:10)) in an example code in the Discourse. Then, a seasoned programmer lectured me that collect() should almost never be used. I asked why and where the API of findall is documented. Then I was shown to the source code!

The current state of documentation seems to be

  1. examples,
  2. the “accurate” description,
  3. source code,
  4. or nothing.

But, what most regular users need is something between 1 and 2: a “helpful” description of the API using an informal language (convention). For example, the A argument of findall is an “indexable collection like a vector or dictionary”, which would be linked to a discussion of generally what datatypes are indexable, where it would be explained that a Range like 1:10 is also indexable.

Here is another example of a different flavor. I was trying to find out how to specify color schemes for contour plots. I googled for “julia Plots contour color schemes” and found the excellent official documentation of color schemes:

https://docs.juliaplots.org/latest/generated/colorschemes/

except that it doesn’t tell you how to use those palette and cgrads objects and the “pre-defined color schemes” with the contour function.

So, I looked at the documentation of the contour function:

https://docs.juliaplots.org/latest/series_types/contour/

The only example it includes is the color=:turbo style. Where are the palette and cgrad objects? Confusingly, I found another example fill = (true, :ranbow) on another website.

The problem here is that there is no API description of the contour function at one place. Or there may be one somewhere, which I haven’t been able to locate.

An example based description is excellent for a tutorial but it’s not adequate as a “manual”.

25 Likes
2

Hm?

help?> findall
search: findall findmax! findmax findlast

  findall(f::Function, A)


  Return a vector I of the indices or keys of A where f(A[I]) returns true. If there are no such elements of A, return an empty array.

  Indices or keys are of the same type as those returned by keys(A) and pairs(A).

  Examples
  ≡≡≡≡≡≡≡≡

(...)

julia> d = Dict(:A => 10, :B => -1, :C => 0)
  Dict{Symbol, Int64} with 3 entries:
    :A => 10
    :B => -1
    :C => 0

  julia> findall(x -> x >= 0, d)
  2-element Vector{Symbol}:
   :A
   :C
1 Like
3

@ryofurue that all sounds reasonable to me. Documenting functions that can have arbitrary methods is hard and we probably don’t do it well enough a lot of the time.

If you have both motivation and a clear vision for improving the documentation, that makes you the best qualified person to start writing the PRs

3 Likes
4

One thing I’ve wondered is if we could add a section header to docstrings something like ## Extended that could hold examples, verbose descriptions, etc. Then when you ?some_function, it would show list of more terse descriptions, but ?some_function -v or ?some_function(::exact, ::input, ::types) (where the types yield a single applicable method) would print the verbose version(s).

If I import DataFrames and then ?vcat, I can scroll for days looking for the one I want.

2 Likes
5

The functionality exists:

julia> """
          f(x)

       My cool function.

       # Extended help

       My really cool function.

       """
       f(x) = 1
f

help?> f
search: f fd for fma fld fld1 fill fdio frexp foldr foldl flush floor float

  f(x)

  My cool function.

  ────────────────────────────────────────────────────────────────────────────

Extended help is available with `??`

help?> ? f
search: f fd for fma fld fld1 fill fdio frexp foldr foldl flush floor float

  f(x)

  My cool function.

  Extended help
  ≡≡≡≡≡≡≡≡≡≡≡≡≡

  My really cool function.

What I think, on the other side, is that it would be nice to have a link pointing to doc pages, when available, where more detailed description of the functions are available, as reading too much information from the REPL is not practical.

7 Likes
6

I share your frustration with the docs, especially when learning about plotting. The contour function is in the API section here but it doesn’t have anything close to full coverage of what is possible. (The one under series_types was only meant to be a tutorial.) There’s a disconnect between people who want to plot something and do very common operations on it like change its color scheme, and trying to understand what things like markercolor::Union{Integer, Symbol, ColorSchemes.ColorScheme, ColorTypes.Colorant} mean here. To be honest I could never get those cgrad and palette objects to work with contours.

I don’t think the structure of the API page that Documenter.jl uses, where all functions are on one long page, is the best way to present the desired information. Matplotlib does it better with a function per page, considering the amount of keywords possible.

5 Likes
7

Thank you for pointing out my error. I’ve corrected the error in my post. My point stands, though: the official document lacks a user-friendly description of “what” A is.

8

I think the idea here is that A may be anything that has keys. Would it be better to provide some concrete examples of possible types in the text?

9

If you have both motivation and a clear vision for improving the documentation, that makes you the best qualified person to start writing the PRs

I have a clear vision, I think, but I don’t have expertise to see whether my vision is a good one or not.

How do you think I should proceed? The following is a long (and not very well-written) discussion of my vision.

All experienced julia programmers have internalized concepts like “iterables” and “indexables”, and so, they naturally expect any “indexable” to work as the A argument of findall(predicate, A).

But, this isn’t necessarily obvious to a newcomer. So, my idea is to explicitly write these “concepts” down as a document and use these concepts in the descriptions of the APIs throughout.

But at this point, the “write your own pull-request” work flow wouldn’t work, because I would need to convince everybody to use these concepts in the documentations.

Each time I ask for a helpful description, the experts’ answer has been that it’s impossible to give an “accurate” description because the API is completely general.

What I’m after is a description using a “convention”, which may not be 100% precise but would act as a helpful guidance when you read the manuals and it would also work as a strong constraint when you write your functions.

To elaborate on the latter point . . . For example, when you write your own function similar to findall(predicate, A), you are strongly encouraged to write it in such a way that any “indexable” works as your A.

Here is an example: Type of array index?
, which is a thread I started more than a year ago. There, some people recommended that I should use Int because it always works as index into an array. But, I ended up writing a function that doesn’t assume that. Then recently I pleasantly realized that my function works for a Dict!

In hindsight, I wrote my function in such a way that any “indexable” works as the argument.

If everybody had known the concept, then that discussion would have been much simpler.

I tentatively call these concepts “type classes”, although I’m not at all sure whether this is a good term or not. The Haskell compiler, for example, enforces at compile time that each argument belongs to the right type classes. The julia language itself doesn’t have such a mechanism, but by convention, the core standard library is designed as if there were type classes. (And I’m sure that the writers of this excellent set of functions had this kind of convention.)

So, the following is a very rough stab at what I have in mind as a document of these concepts:

“Iterable” or “Itarable Collection”: Formally, any object that responds to the iterate() function is an Iterable. When you query an Iterable, it returns one “element” in the collection; when you query it a second time, it returns the next element; and . . . ; when there is no more element, it returns nothing by convention. [Now, this description is not accurate. The function iterate() works differently. My description just provides a “mental model”.]

The for construct uses this property of Iterable. [Are there other standard implicit uses of iterate()?]

For example, a Vector is an Iterable:

# Vector returns 3, then 9, then 4, then 1, and then nothing.
for i in [3, 9, 4, 1]
   println(i)

Likewise, a Range is an Iterable:

# Range returns 2, then 3, . . ., then 9, and finally nothing.
for i in 2:9
   println(i)

Also, a Dict is an Iterable:

# Dict returns ("pi", 3.14), then ("ee", 2.72), . . .
for (k,v) in Dict("pi"=>3.14, "ee"=>2.72, . . . )
   println(k, "=>", v)

You can turn a lot of objects into an Iterable even if they themselves are not Iterables. For example, an input stream of text can be turned into an Iterable by the eachline() function:

istream = open("textfile.txt", "r")
# eachline(istream) returns 1st line, 2nd line, . . . ,
# until the stream is exhausted.
for ll in eachline(istream)
  println(ll)

Internally, eachline() returns an object that responds to the iterate() function.

Analogously, you can create a different Iterable out of an object which is already an Iterable by itself:

# Vector as an Iterable provides its elements one by one:
for x in vec # vec is an Iterable by itself
  println(x)
end
# eachindex creates an Iterable that provides the indices into the vector.
for i in eachindex(vec) # different Iterable
  println(i)

Finally, you can write your own function (like eachline() above) to create an Iterable out of an object, if it makes sense to do so. As an exercise, try writing a function that would return each character from a String:

for c in eachcharacter("this is a sample string.")
  println(c)

[Link to the documentation of iterate().]

Indexable:
Formally, an Indexable is any object A that responds to getindex(A,I).

If you give it a “key”, an Indexable gives you “the corresponding value”.
A prime example is Vector:

v = [3.2, 4.0, 9.5, 1.1]
v[2] # -> 4.0
getindex(v, 2) # equivalent to v[2]

Dict is also an Indexable.
. . . blah blah blah . . .

An Indexable is usually an Iterable. . . . blah blah blah.

6 Likes
10

The problem of vague documentation is to some extent a flip-side of multiple dispatch.

The core of the issue is that Julia does not distinguish very well between the documentation of a function name and the documentation of methods associated with that function name. The way it typically works is that there’s a docstring for the function name when that name is first introduced (as the docstring of some “canonical” method). Subsequent methods can (but often don’t) define their own docstrings, and the docstrings from all methods that are in the methods table at the time the documentation is generated are merged.

The “canonical” docstring for the function name necessarily has to be somewhat vague: The person introducing findall has absolutely no control over the methods of findall that someone else might implement, and what their arguments might be. All they can do is set some reasonable constraints on the arguments and the return values. These are only enforced informally, though.

Ideally, methods that stray too far from the canonical function-name-docstring should define their own docstring to extend the existing documentation – which, in fact, findall does: I’m also seeing docstrings for findall(A) and findall(c::AbstractChar, s::AbstractString) You’ll never have a complete list of all the possible arguments to findall and their properties, though. Apart from the documentation, you might get a lot out of explicitly looking at the methods-table with methods(findall).

There’s always room for improvement for any given documentation. But there’s also the fact that multiple dispatch is a double-edged sword. I would say that your experience as a “newcomer” was to some extent just the learning curve of coming to terms with the dynamic features of Julia.

It’s not just documentation: the exact same issue is at the root of the “traits” discussion. Julia’s design makes strict formal interfaces fundamentally difficult.

6 Likes
11

I tend to find it very helpful when functions are documented in terms of consistency with other existing methods. e.g. findall(x) might be documented to return the same thing as collect(i for (i,v) in enumerate(x) if v)

12

I think the idea here is that A may be anything that has keys . Would it be better to provide some concrete examples of possible types in the text?

I think that’s one way. But, in this particular case, the manual presents a set of related functions. So, perhaps, we want to say, at the top of the document, something like

In this section, A is any indexable object such as array, vector, and range. The point is that all these objects can be indexed into like A[2] or A[“name”] or A[i].

13

That’s the very reason I have been quoting the word “accurate”. If you pursue absolute accuracy, you needs must be vague.

What I’m trying to advocate is a “convention”, a strong guidance, an agreement between developers and programmers. You don’t have to pursue absolute “control”.

That is, we could agree that the A argument of the findall(predicate, A) function should be an “Indexable” and strongly encourage adherence to this convention. We could write up such a convention and encourage everybody to follow it.

Then the documentation of the IPA would be done using the terminology from the convention.

1 Like
14

To some extent, this is a problem in languages with single dispatch too. How most of them seem to handle it is by having a separate docs entry for each method. Does Documenter.jl do the same by default if you use @autodoc or pass it a list of modules and functions? If not, I think a big part of the problem could be addressed by tooling without any language or cultural-level changes (besides adding specific docstrings on more specific methods).

15

Imho, it’s usually confusing and unnecessary when packages extend functions by providing different behavior. If the new interface is different from the original interface, it’s normally better to introduce a new function rather than complicating an existing one.

5 Likes
16

At least in dynamic languages, i.e., with duck-typing, this is indeed similar. I recall that the sklearn docs are full of “array-like of shape (x, y)”.
On the other hand, many languages explicitly distinguish between functions (which have a single implementation only) and methods (which might be defined differently in different classes). Thus, especially in statically typed languages, e.g., Rust or Haskell, it is very clear which interfaces
are used and what is required by them. Here are some examples from the Rust documentation on traits:

// Trait definition, i.e., summarize can have multiple implementations for different types
pub trait Summary {
    fn summarize(&self) -> String;
}

// Generic function, i.e., single implementation
// Uses summarize and thus requires argument type to implement Summary trait
pub fn notify<T: Summary>(item: &T) {
    println!("Breaking news! {}", item.summarize());
}

In contrast, in Julia any function can have multiple implementations (methods) and freely use other generic functions. Thereby blurring the definition and use of an interface.

17

I was thinking of languages such as Java or C#, where the line can likewise be blurred if you have a bunch of method overloads that aren’t overriding some abstract method. They seem to get along well enough with their docs systems despite this, so I don’t see any reason Julia couldn’t do the same. For packages that overload methods in other packages/stdlibs, maybe something like the impl X for Y rendering used by Rustdoc would make sense. In other words, showing the source module of the function as well.

1 Like
18

This is just my personal opinion about one aspect of the documentation that I think could be improved upon: the examples used. Sometimes they’re more complicated than they should.

For instance, let’s take the documentation of broadcasting.

First, not sure why using a = rand(2,1); A = rand(2,3); as the simplest example to illustrate what broadcasting does. In general, I see a tendency to use matrix as a first example, which I think it’s not the best approach.

I’d consider it as a first example (cases with matrices and others can be added after that).

x = [1,2,3]
y = [10,10,10]

broadcast(*, x, y)

to show that it’s unnecessary, and then compare it with

x = [1,2,3]
y = 10

broadcast(*, x, y)

Second, examples sometimes use functions that are quite specific and/or require additional notions. In the same section, I wouldn’t have used ceil.(UInt8, [1.2 3.4; 5.6 6.7]). It requires that the user know swhat ceil does and what Uint8 is, when the documentation should start with an example that covers the whole audience.

Third, I’d also avoid defining collections within the arguments of a function. The example of ceil is one case. Another is:
([1, 2, 3], [4, 5, 6]) .+ ([1, 2, 3],)
which requires rereading it twice to separate the brackets and see what’s going on.

Alternative:

x = ([1, 2, 3], [4, 5, 6])
y = ([1, 2, 3],)

x .+ y

and again, I would’ve probably used a simpler example for making the point.

I focused on broadcasting, but it’s just a random example of a common pattern.

5 Likes
19

While I agree that a lot of documentation could be improved (this is true for almost all software), note that discussions like this are unlikely to accomplish a significant change because they are not specific enough.

It is better to focus effort on actually fixing the docs whenever you encounter a problem. Eg PR that reviews the docs for a group of methods and clarifies these questions.

20

Please look at my response to @Raf (I think. I’m sorry if I’m mistaken), who gave me the same advice.

In short, my vision is to write up a document, which would define type classes as convention (because the language itself doesn’t have a mechanism to enforce type classes). Then we would use the terms and ideas in the document to revise the documentations.

In that post of mine, I just presented a sample of the document I have in mind, but I don’t have expertise enough to complete a draft of such a big document.

So, to accomplish what I think we can/should do, just the write-your-PR approach doesn’t work.