Why is there a lack of advanced metaprogramming documentation?

Community
1

While Julia has a good metaprogramming starter guide that covers AST manipulation, too much metaprogramming remains undocumented or poorly documented. For example, how could gradient(f,x) work? You would need to inspect the function. How would you do that? What if you want to do a compiler extension or custom compilation pass? You’re often left to figure it out on your own. Some might say that very few people would use it. However, I don’t think that’s a good justification for not documenting such a thing.

  1. One of Julia’s philosophies is that you should be able to do anything if you’re good enough. Leaving some paths that nerds can dive through would be a good idea. They’re likely to later on make disproportionate contributions to the ecosystem.

  2. Without documentation establishing the best practice of how to do certain things, some would end up implementing them in hacky ways, perhaps relying unnecessarily on compiler internal and so on.

But then again, some metaprogramming features might not be in the base Julia, but rather in some external packages like Cassette. So, perhaps the person best suited to take on this task is not the Julia team but rather the developers of those packages.

But then, how did these packages get implemented in the first place?

There be dragons.

Or perhaps these features are not yet officially supported but some clever people found a way to hack these features into Julia? Or maybe they’re internally supported for internal uses but not yet ready for public use only to find out some packages use them anyway? Maybe exposing these features would have the users encounter bugs that won’t surface under normal uses?

What on earth is going on here? Is there a plan? If the plan is that these features are reserved for advanced users only, that is a valid plan. However, currently, there doesn’t seem to be a documented plan. There seems to only be magic. There is no indication if or when Julia will officially support these features. Meanwhile, some packages are already using these features!

What is going on here? I am confused. I appreciate these packages that they exist, but I am unable to figure out what sorcery are they doing.

And again, I reiterate that it’s not the point that only advanced users should use these features. You can put in disclaimers. The point is that there doesn’t seem to be any documentation that these features exist at all.

2

It seems you’re looking for developer documentation. Compiler extensions are that kind of thing. There is a developer section in the manual: Julia Documentation · The Julia Language

Your example with gradient(f, x) does not require inspecting the function. E.g. the gradient in ForwardDiff uses dual numbers (i.e. it just calls your function with a variable of type Vector{ForwardDiff.Dual{...}}). How ForwardDiff Works · ForwardDiff

using ForwardDiff: gradient

function f(x)
    println(typeof(x))
    sin(prod(x))
end

gradient(f, [0.3, 0.2])

yields output:

Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(f), Float64}, Float64, 2}}
2-element Vector{Float64}:
 0.19964010798704085
 0.29946016198056125
2 Likes
3

So I try and make a point about talking about this kinda thing at juliacon.
I think I counted and like half my juliacon talks were about some nightmarish thing you could do with generalized metaprogramming.
Tricks.jl, ExprTools.jl, MagneticReadhead.jl, and how they work.
I know a few other people do similar.
(I think it is good to have this kinda “advanced” talk for people who have seen everything)

I am giving another this year “Plundering Core.Compiler: Adventures in IR transformations”
that is aimed explictly in telling people how to use the compiler’s own internal tools for doing compiler transforms.

I imagine reason none of this is this is documented is because it is is all internal and unstable. And there is a general belief that once you document something then changing it is breaking.

10 Likes
4

These two assertions you make are in direct conflict with each other:

Compiler extensions and custom compilation passes are the actual dragons themselves. They are hacky. They are by definition relying on compiler internals. Even when done in packages. They’re hard to maintain, because they are hacks.

The answer to many "why"s like this are simply that it is the way it is because it is that way. I’ve seen that compiler devs have prioritized the docs that help onboard more compiler devs, and they’re written targeting that audience. Documentation is work, both in creation and maintenance. But of course the status quo can always change, by you or others. If you’re interested in more of this, you can check out this draft PR: Support execution of code from external abstract interpreters by vchuravy · Pull Request #52964 · JuliaLang/julia · GitHub.

6 Likes
5

Okay. I think I understand now. It seems the language has some internal features that can be unstable and undocumented, but get used anyway. It’s normal that you start your journey seeing things beautiful and pristine, but the longer and deeper you go, you start seeing the cracks. Compiler extensions/etc will definitely depend on the compiler internals more than regular packages, but they don’t necessarily need to rely on every minute detail of the compiler implementation. Hopefully one day (maybe 5+ years), the compiler internal will stabilize enough that you can write a stable compiler extension/etc. Is the current system a bit ugly? Maybe. But it is necessary, at least for now. I’m not saying this to disparage the devs of Julia or any package by any means. You’re awesome!

I am confident the Julia team is working on solving this problem. Good luck with that. Thank you for all the awesome works all these years.