I think that Julia documentation needs to be more “discoverable” or at
least “explorable”. I would like to have a local, searchable set of
documentation that you could install for the local Julia covering all
installed packages and searchable whether or not you have already
loaded the modules
TL;DR follows
It is often difficult to find a package that does something (algorithm,
solver, graphics,…) that is needed because there doesn’t seem to be a
simple way to search through each one-by-one from their repository to
their documentation.
The alternative is to add the package, use the package, and then try to
search the docs from the REPL or your IDE.
I propose that all documentation for packages for a given .julia
installation have a local install which can be searched for anything
in any package on the system.
In MATLAB you can search the documentation of application and across
all installed toolboxes.
The Perl Data Language adds the full documentation of each installed
PDL package to a searchable database to support a pdldoc command.
( E.g. apropos xxx will return the short description of all matches for ```xxx``
in the given PDL installation and give the package name or function name.)
Some thoughts on possible implementations:
Add local documentation support to Pkg to support searches across
multiple versions and for reproducibility
Enhance the REPL help command with better fuzzy matching, maybe help for exact matches (case insensitive) and improve the apropos
command with smarter matching and result presentation.
Provide a local web interface to present multi-media information
and support use of the more sophisticated generated docs
Benefits could be:
Rapid, global search to understand and find packages of interest
and descriptions of functions and methods (especially those that
are already installed—multiple dependencies can pull in multple
packages that you may still need to understand to use the top level
```add-ed`` package!)
Local installed documentation with a web interface would improve
the general quality of Julia documentation since every package
would have “web page docs”
Local docs search could improve the odds that someone can find
and use the appropriate packages
Developers might get instant feedback from new users to improve
the content for all (there could even be some sort of widget tool to
generate the PR for sending)
A next-generation capability could be to have a way to process the
local documentation with an OpenAI enhanced interface. We could have
the ability to search for documentation from the REPL or anywhere with
queries like
help?> Are there any packages to do least squares fits with complex numbers?
I started work on something along those lines a while ago here, but I never really made much progress because the basic doc infrastructure in Julia isn’t really setup right now to create a version of docs that one could easily consume locally. But I agree, that would be a great feature! If folks want to discuss this more, would probably make sense to break that thread out of this general thread into its own topic.
Why locally? If there was a searchable repository for all available packages (I guess essentially that already exists), what we need is a package that filters the search automatically for the currently loaded Julia packages of a section / environment. Maybe that’s easier to develop. (having the docs built locally seems a waste of resources)
I mean, this can first be a web interface (on JuliaHub, for example) that allows searching multiple docs at once filtered by sets of packages of interest, to later have some simple tool to bind that directly to a Julia section.
Selecting a few packages there produces a unique URL that could easily be modified for new search queries if you want a non-browser based interface to making the query
Juliahub has Ask AI and in your example the (truncated) output would be. I guess copilot chat would be similar (from vscode) and not the repl.
Yes, there are packages in Julia that can perform least squares fits with complex numbers. One such package is the LsqFit.jl package. It provides basic least-squares fitting in pure Julia and supports complex-valued data and models.
To use LsqFit.jl for complex least squares fits, you can define your model function to accept and return complex numbers. Then, you can pass complex-valued data and initial parameter values to the curve_fit function provided by LsqFit.jl.
Here’s an example of how you can use LsqFit.jl for complex least squares fits:
using LsqFit
# Define your model function that accepts complex numbers
function model_func(x, p)
a, b = p
return a * x + b
end
(truncated)
My point was that I believe it is difficult to find packages that are relevant
to your needs with the current “documentation as a service”" approach.
Making the documentation more easily searchable with less hurdles to
jump (i.e. Load the package first and then you can use REPL help. Wait!
I don’t know what package I need/I don’t have xxxx installed/…) would be
a win, especially for beginners.
I believe that local documentation capabilities would, at the least, make
the existing already produced documentation more effective and usable.
Modules that already generate web site documentation would just need
common standards and framework to install locally. The content could
be built when the package is generated. Distribution would come with the
standard Pkg.
Yes, the Juliahub search looks nice. I think it would be even better
with a “Juliahub at the edge” interface!
Having a local documentation install would be a big help for situations
where there is no internet access available, allowed, or just so slow you
might as well print out the source code and read the doc strings by eye.
I would pay for that as a sponsored version of the vscode extension, especially if it ran all of the Documenter.jl output for installed packages through a process to generate local quarto pages with a consistent look.
In my ideal world, the package manger would not just download the package itself, but also a built markdown version of each package that you install and store that in a known location in the Julia depot. The Julia VS Code extension could then show a tree in the UI that shows a merged content tree of all the docs for all the packages in your current environment. I think the main UI issue I have right now is that when I work in VS Code, there is machine readable info available that is a very strong indicator which docs I might be interested right now (namely the stuff that is in my active Project.toml), and I’d like to have a UI where that is used without much context switching.
I think you are describing how the VScode rust analyzer works with Cargo.toml You hover over a dependency, its shows a list of versions and a link to the documentation for each version, but the documentation is remote. Or with rust rover it should be similar: link
For what it’s worth, that is literally what DocumentationGenerator.jl does (JuliaHub uses it under the hood).
However, people do all kinds of weird stuff with their docs, and sometimes they’re just super expensive to build, so it is very hard to reliably build them for arbitrary packages. And, paradoxically, it’s usually the more used and better-maintained packages that are the most problematic (I’d say because they have the resources to write more complex docs).
But I think it basically boils down to what David said that
basic doc infrastructure in Julia isn’t really setup right now to create a version of docs that one could easily consume locally.
We should probably think about having a more standardized architecture for setting up simple docs for a package.
Looking at how Julia 1.11 is going with Pkg.jl out of the system image, I do think we are going to need some additional command line drivers like jlpkg perhaps with their own system images.
A utility system image with a driver with cargo like functionality including packages such as Pkg.jl, Documenter.jl, and DocumentationGenerator.jl among other developer utlities might be nice to see.
And, paradoxically, it’s usually the more used and better-maintained packages that are the most problematic
I feel seen.
I think that rather than VSCode trying to build the docs from Markdown, a much simpler approach is for them to consume the HTML tarball that is already built by Documenter. In most cases, this is in the gh-pages branch, and it has a standard structure.
In fairness, also on SciML and some plotting packages etc. But this is obviously not meant to be a criticism – good docs for complex software are themselves complex, and I think it’s fine if those need a bit of special handling.
Taking advantage of hosted documentation tarballs for such packages is also something we probably want to do. But I also think that there is value in simplifying documentation building in Julia to the point where you can do things automatically, and it just works. I opened an issue for one idea:
Now I’m thinking about how javadoc works. Also, maven has a nice way of deploying jar files with documentation artifacts and downloading those artifacts.
One way to distribute the documentation could be by adding tarballs to GitHub releases. That would be a pretty standard place to host such artifacts, and they could be uploaded when Documenter runs on the tag. This might also get around the problem of old docs taking up too much space on gh-pages etc., without fully deleting them.
I don’t think that would really work, the HTML built by Documenter is designed to be viewed “standalone” in a web browser. In the VS Code extension we for example wouldn’t want the TOC as part of the web page, instead we would want to view that as a tree in the VS Code UI. The ideal thing for us would just be a md file for each help page, and then we can (on the fly) convert that into the kind of HTML that makes most sense within VS Code.
I guess that could work, but there are some weird timing issues then, for example packages typically appear in the registry before the github release is created.
I think in my ideal world, as part of package registration a markdown version of the docs gets built (via a very standardized entry point), and then they get hosted/stored as part of the package server infrastructure.
We actually made some progress on this over at VS Code doc hosting integration · Issue #10 · MichaelHatherly/Publish.jl · GitHub a few years ago, but not clear to me whether that package is still being worked on? The key other thing that we would need is something like a JSON file with the actual TOC for a given package’s docs.
Makie migrated away from Documenter a while ago because it did not really offer the kind of flexibility I wanted at the time. That’s fine, it’s an opinionated tool. Makie has special demands due to its heavy focus on visuals.
So as we’re probably not going to move back to Documenter any time soon, I’m interested in solutions that also work for packages with their own special structure. Our markdown files have special Franklin commands in them so you can’t just ingest them, but maybe we could build to an intermediate markdown format?
on SciML and some plotting packages etc. But this is obviously not meant to be a criticism – good docs for complex software are themselves complex, and I think it’s fine if those need a bit of special handling.
this PR: