# Documentation needs to be more discoverable and explorable

**URL:** <https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549>\
**Category:** General Usage\
**Tags:** documentation\
**Created:** [November 21, 2023, 3:26pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549 "2023-11-21T15:26:09Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![devel-chm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devel-chm/32/3572_2.png) [@devel-chm](https://discourse.julialang.org/u/devel-chm)\
**Post date:** [November 21, 2023, 3:26pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/1 "2023-11-21T15:26:09Z")

</div>

I think that Julia documentation needs to be more “discoverable” or at  
least “explorable”. I would like to have a **local, searchable** set of  
documentation that you could install for the local Julia covering all  
installed packages and searchable whether or not you have already  
loaded the modules

TL;DR follows

It is often difficult to find a package that does something (algorithm,  
solver, graphics,…) that is needed because there doesn’t seem to be a  
simple way to search through each one-by-one from their repository to  
their documentation.

The alternative is to add the package, use the package, and then try to  
search the docs from the REPL or your IDE.

I propose that all documentation for packages for a given `.julia`  
installation have a local install which can be searched for anything  
in any package on the system.

In MATLAB you can search the documentation of application and across  
all installed toolboxes.

[The Perl Data Language](https://pdl.perl.org/) adds the full documentation of each installed  
PDL package to a searchable database to support a `pdldoc` command.  
( E.g. `apropos xxx` will return the short description of all matches for ```xxx``  
in the given PDL installation and give the package name or function name.)

Some thoughts on possible implementations:

- Add local documentation support to Pkg to support searches across  
multiple versions and for reproducibility
- Enhance the REPL help command with better fuzzy matching, maybe  
`help` for exact matches (case insensitive) and improve the `apropos`  
command with smarter matching and result presentation.
- Provide a local web interface to present multi-media information  
and support use of the more sophisticated generated docs

Benefits could be:

- Rapid, global search to understand and find packages of interest  
and descriptions of functions and methods (especially those that  
are already installed—multiple dependencies can pull in multple  
packages that you may still need to understand to use the top level  
```add-ed`` package!)
- Local installed documentation with a web interface would improve  
the general quality of Julia documentation since _every_ package  
would have _“web page docs”_
- Local docs search could improve the odds that someone can find  
and use the appropriate packages
- Developers might get instant feedback from new users to improve  
the content for all (there could even be some sort of widget tool to  
generate the PR for sending)

A next-generation capability could be to have a way to process the  
local documentation with an OpenAI enhanced interface. We could have  
the ability to search for documentation from the REPL or anywhere with  
queries like

```
help?> Are there any packages to do least squares fits with complex numbers?

```

How cool would that be!!

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [November 21, 2023, 8:00pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/2 "2023-11-21T20:00:04Z")

</div>

> [@devel-chm](#):
>
> I would like to have a **local, searchable** set of  
> documentation that you could install for the local Julia covering all  
> installed packages and searchable whether or not you have already  
> loaded the modules

I started work on something along those lines a while ago [here](https://github.com/julia-vscode/julia-vscode/pull/2230), but I never really made much progress because the basic doc infrastructure in Julia isn’t really setup right now to create a version of docs that one could easily consume locally. But I agree, that would be a great feature! If folks want to discuss this more, would probably make sense to break that thread out of this general thread into its own topic.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [November 21, 2023, 8:15pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/3 "2023-11-21T20:15:54Z")

</div>

> [@davidanthoff](#):
>
> one could easily consume locally.

Why locally? If there was a searchable repository for all available packages (I guess essentially that already exists), what we need is a package that filters the search automatically for the currently loaded Julia packages of a section / environment. Maybe that’s easier to develop. (having the docs built locally seems a waste of resources)

I mean, this can first be a web interface (on JuliaHub, for example) that allows searching multiple docs at once filtered by sets of packages of interest, to later have some simple tool to bind that directly to a Julia section.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [November 22, 2023, 7:02am UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/4 "2023-11-22T07:02:51Z")

</div>

Juliahub does offer this 🙂

[https://juliahub.com/ui/Search?q=fft&type=packages](https://juliahub.com/ui/Search?q=fft&type=packages)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/4/5/453dd59720a3c339d6120ad14bc3ec0ec3b0f68a.png)

Under “documentation search”, there’s a filter option to only search in certain packages:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/b/7/b7e6d51d31a1f85841e03776583498e333d68178.png)  
Selecting a few packages there produces a unique URL that could easily be modified for new search queries if you want a non-browser based interface to making the query

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [November 22, 2023, 1:21pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/5 "2023-11-22T13:21:31Z")

</div>

Juliahub has [Ask AI](https://juliahub.com/ui/AskAI) and in your example the (truncated) output would be. I guess copilot chat would be similar (from vscode) and not the repl.

> Yes, there are packages in Julia that can perform least squares fits with complex numbers. One such package is the `LsqFit.jl` package. It provides basic least-squares fitting in pure Julia and supports complex-valued data and models.
> 
> To use `LsqFit.jl` for complex least squares fits, you can define your model function to accept and return complex numbers. Then, you can pass complex-valued data and initial parameter values to the `curve_fit` function provided by `LsqFit.jl`.
> 
> Here’s an example of how you can use `LsqFit.jl` for complex least squares fits:
> 
> ```julia
> using LsqFit
> 
> # Define your model function that accepts complex numbers
> function model_func(x, p)
> a, b = p
> return a * x + b
> end
> 
> (truncated)
> 
> ```

```julia

```

---

<div class="post-metadata">

**Author:** ![devel-chm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devel-chm/32/3572_2.png) [@devel-chm](https://discourse.julialang.org/u/devel-chm)\
**Post date:** [November 22, 2023, 2:01pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/6 "2023-11-22T14:01:13Z")

</div>

My point was that I believe it is difficult to find packages that are relevant  
to your needs with the current “documentation as a service”" approach.

Making the documentation more easily searchable with less hurdles to  
jump (i.e. Load the package _first_ and _then_ you can use REPL help. Wait!  
I don’t know what package I need/I don’t have xxxx installed/…) would be  
a win, especially for beginners.

I believe that local documentation capabilities would, at the least, make  
the existing _already produced documentation_ more effective and usable.

Modules that already generate web site documentation would just need  
common standards and framework to install locally. The content could  
be built when the package is generated. Distribution would come with the  
standard Pkg.

---

<div class="post-metadata">

**Author:** ![devel-chm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/devel-chm/32/3572_2.png) [@devel-chm](https://discourse.julialang.org/u/devel-chm)\
**Post date:** [November 22, 2023, 2:06pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/7 "2023-11-22T14:06:05Z")

</div>

Yes, the Juliahub search looks nice. I think it would be even better  
with a “Juliahub at the edge” interface!

> [@baggepinnen](#):
>
> Juliahub does offer this 🙂

Having a local documentation install would be a big help for situations  
where there is no internet access available, allowed, or just so slow you  
might as well print out the source code and read the doc strings by eye. 🙂

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [November 22, 2023, 3:01pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/8 "2023-11-22T15:01:22Z")

</div>

I would pay for that as a sponsored version of the vscode extension, especially if it ran all of the Documenter.jl output for installed packages through a process to generate local quarto pages with a consistent look.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [November 22, 2023, 7:48pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/9 "2023-11-22T19:48:41Z")

</div>

> [@lmiq](#):
>
> Why locally?

I often work on planes 🙂

In my ideal world, the package manger would not just download the package itself, but also a built markdown version of each package that you install and store that in a known location in the Julia depot. The Julia VS Code extension could then show a tree in the UI that shows a merged content tree of all the docs for all the packages in your current environment. I think the main UI issue I have right now is that when I work in VS Code, there is machine readable info available that is a _very_ strong indicator which docs I might be interested right now (namely the stuff that is in my active `Project.toml`), and I’d like to have a UI where that is used without much context switching.

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [November 22, 2023, 8:56pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/10 "2023-11-22T20:56:18Z")

</div>

I think you are describing how the VScode rust analyzer works with `Cargo.toml` You hover over a dependency, its shows a list of versions and a link to the documentation for each version, but the documentation is remote. Or with rust rover it should be similar: [link](https://www.jetbrains.com/help/rust/2023.2/toml-files.html#view-crate-docs)

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [November 22, 2023, 10:50pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/11 "2023-11-22T22:50:19Z")

</div>

For Julia, the manual is there.

```julia
~/julia-1.9.3/share/doc/julia/html/en# ls                    
NEWS.html base index.html search.html stdlib
assets devdocs manual search_index.js

```

For packages, what is stopping us from running Documenter.jl on the packages in .julia/packages ?

---

<div class="post-metadata">

**Author:** ![mortenpi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mortenpi/32/158_2.png) [@mortenpi](https://discourse.julialang.org/u/mortenpi)\
**Post date:** [November 22, 2023, 11:34pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/12 "2023-11-22T23:34:45Z")

</div>

> [@davidanthoff](#):
>
> In my ideal world, the package manger would not just download the package itself, but also a built markdown version of each package that you install and store that in a known location in the Julia depot.

> [@mkitti](#):
>
> For packages, what is stopping us from running [Documenter.jl](https://juliahub.com/ui/Packages/Documenter) on the packages in .julia/packages ?

For what it’s worth, that is literally what DocumentationGenerator.jl does (JuliaHub uses it under the hood).

However, people do all kinds of weird stuff with their docs, and sometimes they’re just super expensive to build, so it is _very_ hard to reliably build them for arbitrary packages. And, paradoxically, it’s usually the more used and better-maintained packages that are the most problematic (I’d say because they have the resources to write more complex docs).

But I think it basically boils down to what David said that

> basic doc infrastructure in Julia isn’t really setup right now to create a version of docs that one could easily consume locally.

We should probably think about having a more standardized architecture for setting up simple docs for a package.

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [November 23, 2023, 12:06am UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/13 "2023-11-23T00:06:07Z")

</div>

Something like [`cargo doc`](https://doc.rust-lang.org/cargo/commands/cargo-doc.html) would be really nice.

I added a reference to this thread to the list of future discussion topics of the [documentation working group](https://discourse.julialang.org/t/documentation-oriented-community-calls/104887).

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [November 23, 2023, 12:27am UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/14 "2023-11-23T00:27:57Z")

</div>

Looking at how Julia 1.11 is going with Pkg.jl out of the system image, I do think we are going to need some additional command line drivers like [`jlpkg`](https://github.com/fredrikekre/jlpkg) perhaps with their own system images.

A utility system image with a driver with `cargo` like functionality including packages such as Pkg.jl, Documenter.jl, and DocumentationGenerator.jl among other developer utlities might be nice to see.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [November 23, 2023, 12:51am UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/15 "2023-11-23T00:51:12Z")

</div>

> And, paradoxically, it’s usually the more used and better-maintained packages that are the most problematic

I feel seen.

I think that rather than VSCode trying to build the docs from Markdown, a much simpler approach is for them to consume the HTML tarball that is already built by Documenter. In most cases, this is in the `gh-pages` branch, and it has a standard structure.

---

<div class="post-metadata">

**Author:** ![mortenpi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mortenpi/32/158_2.png) [@mortenpi](https://discourse.julialang.org/u/mortenpi)\
**Post date:** [November 23, 2023, 1:00am UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/16 "2023-11-23T01:00:37Z")

</div>

> [@odow](#):
>
> I feel seen.

😅

In fairness, also 👀 on SciML and some plotting packages etc. But this is obviously not meant to be a criticism – good docs for complex software are themselves complex, and I think it’s fine if those need a bit of special handling.

Taking advantage of hosted documentation tarballs for such packages is also something we probably want to do. But I also think that there is value in simplifying documentation building in Julia to the point where you _can_ do things automatically, and it _just works_. I opened an issue for one idea:

> <https://github.com/JuliaDocs/Documenter.jl/issues/2353>
>
> We probably want to move towards a more standardized and reliable docs build met…hod. So my thinking right now is turning \`docs/make.jl\` just into a single Documenter command:
> 
> \`\`\`julia
> Documenter.build(package\_directory)
> \`\`\`
> 
> It would then figure out some reasonable documentation build for that package environment. Key is that \`package\_directory\` must be a valid package-type environment.
> 
> \- First MVP would be something that just builds maybe the README + autogenerated list of (public) docstrings.
> \- Second iteration would automatically pick up \`.md\` files from the \`docs/\` directory, allowing for a proper manual to be built.
> \- Next step would be to pick up some (limited) config options from the project TOML files, for customization (https://github.com/JuliaDocs/Documenter.jl/issues/1350).
> 
> We should also create a GitHub Action, which instantiates the environment and calls this command. This way setting up docs for a package would become much simpler for the user (deployment side should also come in somewhere them, but \`Documenter.deploy()\` is available, and we can have enough metadata in \`siteinfo.js\` to determine the necessary \`deploydocs\` arguments I think).
> 
> A little bit unsure how to exactly handle environments here. Should there be a \`docs/Project.toml\`, that has Documenter? We definitely need to be able to \`Pkg.add\` Documenter together with the package though.
> 
> Note: this would not be for everyone. More complex manuals would still use custom \`make.jl\` setups. But I would hope that this would simplify the docs life for like 80% of packages.
> 
> Also some related discussion on Discourse: https://discourse.julialang.org/t/julias-documentation-needs-to-be-more-discoverable-and-explorable/106549

> [@mkitti](#):
>
> A utility system image with a driver with `cargo` like functionality including packages such as [Pkg.jl](https://juliahub.com/ui/Packages/Pkg), [Documenter.jl](https://juliahub.com/ui/Packages/Documenter), and [DocumentationGenerator.jl](https://juliahub.com/ui/Packages/DocumentationGenerator) among other developer utlities might be nice to see.

Tangent, but I ❤ this PR:

> <https://github.com/JuliaLang/julia/pull/52103>
>
> This aims to bring similar functionality to Julia as the \`-m\` flag for Python wh…ich exists to directly run some function in a package and being able to pass arguments to that function.
> 
> While in Python, \`python -m package args\` runs the file \`\<package\>.\_\_main\_\_.py\`, the equivalent Julia command (\`julia -m Package args\`) instead runs \`\<Package\>.main(args)\`. A package can also have a \`main\` function in a submodule runnable with \`julia -m Package.SubModule args\`. The package is assumed to be installed in the environment \`julia\` is run in.
> 
> An example usage could be:
> 
> Add the package:
> \`\`\`julia
> (@v1.11) pkg\> add https://github.com/KristofferC/Rot13.jl
> Cloning git-repo \`https://github.com/KristofferC/Rot13.jl\`
> Updating git-repo \`https://github.com/KristofferC/Rot13.jl\`
> Resolving package versions...
> Updating \`~/.julia/environments/v1.11/Project.toml\`
> \[43ef800a\] + Rot13 v0.1.0 \`https://github.com/KristofferC/Rot13.jl#master\`
> Updating \`~/.julia/environments/v1.11/Manifest.toml\`
> \[43ef800a\] + Rot13 v0.1.0 \`https://github.com/KristofferC/Rot13.jl#master\`
> \`\`\`
> 
> And then it can be run (since it has a \`main\` function) via:
> 
> \`\`\`
> ❯ ./julia/julia -m Rot13 "encrypt this for me" "and this as well"
> rapelcg guvf sbe zr
> naq guvf nf jryy
> \`\`\`
> 
> I'm not sure if \`-m/--module\` is the best choice but perhaps the association to Python makes it worth it.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [November 23, 2023, 1:17am UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/17 "2023-11-23T01:17:14Z")

</div>

Now I’m thinking about how [`javadoc`](https://docs.oracle.com/en/java/javase/11/javadoc/javadoc-command.html) works. Also, `maven` has a nice way of deploying jar files with documentation artifacts and downloading those artifacts.

[https://maven.apache.org/plugins/maven-deploy-plugin/examples/deploying-sources-javadoc.html](https://maven.apache.org/plugins/maven-deploy-plugin/examples/deploying-sources-javadoc.html)

---

<div class="post-metadata">

**Author:** ![mortenpi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mortenpi/32/158_2.png) [@mortenpi](https://discourse.julialang.org/u/mortenpi)\
**Post date:** [November 23, 2023, 1:35am UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/18 "2023-11-23T01:35:21Z")

</div>

One way to distribute the documentation could be by adding tarballs to GitHub releases. That would be a pretty standard place to host such artifacts, and they could be uploaded when Documenter runs on the tag. This might also get around the problem of old docs taking up too much space on `gh-pages` etc., without fully deleting them.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [November 23, 2023, 6:41pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/19 "2023-11-23T18:41:45Z")

</div>

> [@odow](#):
>
> I think that rather than VSCode trying to build the docs from Markdown, a much simpler approach is for them to consume the HTML tarball that is already built by Documenter.

I don’t think that would really work, the HTML built by Documenter is designed to be viewed “standalone” in a web browser. In the VS Code extension we for example wouldn’t want the TOC as part of the web page, instead we would want to view that as a tree in the VS Code UI. The ideal thing for us would just be a md file for each help page, and then we can (on the fly) convert that into the kind of HTML that makes most sense within VS Code.

> [@mortenpi](#):
>
> One way to distribute the documentation could be by adding tarballs to GitHub releases.

I guess that could work, but there are some weird timing issues then, for example packages typically appear in the registry before the github release is created.

I think in my ideal world, as part of package registration a markdown version of the docs gets built (via a very standardized entry point), and then they get hosted/stored as part of the package server infrastructure.

We actually made some progress on this over at [VS Code doc hosting integration · Issue #10 · MichaelHatherly/Publish.jl · GitHub](https://github.com/MichaelHatherly/Publish.jl/issues/10) a few years ago, but not clear to me whether that package is still being worked on? The key other thing that we would need is something like a JSON file with the actual TOC for a given package’s docs.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 23, 2023, 6:47pm UTC](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549/20 "2023-11-23T18:47:03Z")

</div>

Makie migrated away from Documenter a while ago because it did not really offer the kind of flexibility I wanted at the time. That’s fine, it’s an opinionated tool. Makie has special demands due to its heavy focus on visuals.

So as we’re probably not going to move back to Documenter any time soon, I’m interested in solutions that also work for packages with their own special structure. Our markdown files have special Franklin commands in them so you can’t just ingest them, but maybe we could build to an intermediate markdown format?

[Next page](https://discourse.julialang.org/t/documentation-needs-to-be-more-discoverable-and-explorable/106549.md?page=2)