[ANN] DispatchDoctor.jl 🩺 – offers you a prescription for type stability

Announcing…

DispatchDoctor :stethoscope:

The doctor’s orders: no type instability allowed!

Dev Build Status Coverage Aqua QA

This experimental package provides the @stable macro to enforce that functions have type stable return values.

using DispatchDoctor: @stable

@stable function relu(x)
    if x > 0
        return x
    else
        return 0.0
    end
end

Calling this function will throw an error for any type instability:

julia> relu(1.0)
1.0

julia> relu(0)
ERROR: TypeInstabilityError: Instability detected in function `relu`
with arguments `(Int64,)`. Inferred to be `Union{Float64, Int64}`,
which is not a concrete type.

Code which is type stable should compile away the check:

julia> @stable f(x) = x;

with @code_llvm f(1):

define i64 @julia_f_12055(i64 signext %"x::Int64") #0 {
top:
  ret i64 %"x::Int64"
}

Meaning there is zero overhead on this particular type stability check. (Please post an issue for any known examples of this not being the case!)

You can also use @stable on entire blocks of code, including begin-end blocks, module, and anonymous functions – it will propagate through everything and “stabilize” all functions. The inverse of @stable is @unstable which toggles it back off.

@stable begin
    f() = rand(Bool) ? 0 : 1.0

    module A
        # Will apply to code inside modules:
        g(; a, b) = a + b

        # Will recursively apply to included files:
        include("myfile.jl")
        module B # as well as nested submodules!

            # `@unstable` inverts `@stable`:
            using DispatchDoctor: @unstable
            @unstable h() = rand(Bool) ? 0 : 1.0

            # This can also apply to code blocks:
            @unstable begin
                h(x::Int) = rand(Bool) ? 0 : 1.0
                # ^ And target specific methods
            end
        end
    end
end

All methods in the block will be wrapped with the type stability check:

julia> f()
ERROR: TypeInstabilityError: Instability detected in function `f`.
Inferred to be `Union{Float64, Int64}`, which is not a concrete type.

(Tip: in the REPL, you must wrap modules with @eval, because the REPL has special handling of the module keyword.)

You can disable stability errors for a single scope with the allow_unstable context:

julia> @stable f(x) = x > 0 ? x : 0.0

julia> allow_unstable() do
           f(1)
       end
1

although this will error if you try to use it simultaneously from two separate threads.

Note that instability errors are also automatically skipped during precompilation.

Note!

@stable will have no effect on code if it is:

  • Within an @unstable block
  • Within a macro definition
  • A function inside another function (i.e., a closure) – although the wrapping function will still get checked (see this example)
  • A generated function
  • Within an @eval statement
  • Within a quote block
  • If the function name is an expression (such as parameterized functions like MyType{T}(args...) = ...)

You can safely use @stable to wrap any of these cases; they will simply be ignored. Although, if you use @stable internally in any of these cases, (like calling @stable directly on a closure), then it will apply.

And of course if you find other cases of @stable not playing well with certain Julia features, please raise an issue.

Also, @stable is a no-op in unsupported Julia versions (before 1.10) (and also any future Julia versions I haven’t tested yet)

Credits

Many thanks to @Elrod, @thofma, @matthias314 and others for tips on Improving speed of runtime dispatch detector

83 Likes

For me this is the most common source of type instabilities since my functional style of programming includes a lot of closures.

Is it planned in future? As long 15276 is open, this would be a killer feature for me.

5 Likes

It should already work for many cases. What I meant was that in the case of

@stable function f(x)
    function inner()
        x
    end
    inner()
end

You would stabilize f, but not inner. So if f has a type instability resulting from inner, that would still be caught!

If you meant: can you also have it check the output of inner, then yes, you can do

@stable function f(x)
    @stable function inner()
        x
    end
    inner()
end

And it will check both that the output of inner is type stable, as well as the output of f.

It basically just doesn’t propagate @stable into closures, if that makes sense. But I could certainly reconsider this. I just didn’t want it to start getting applied to all the little closures in things like map(x -> x^2, ar) which would IMO be overkill. But maybe people want that?

6 Likes

For example:

julia> @stable function f(x)
           function inner()
               x = x
               x
           end
           inner()
       end
f (generic function with 1 method)

julia> f(1)
ERROR: TypeInstabilityError: Instability detected in `f`
defined at REPL[4]:1 with arguments `(Int64,)`. Inferred
to be `Any`, which is not a concrete type.
1 Like

Hm, I think this isn’t a particularly strong counter-argument, because, at least in my experience, those “one liner” closures are 99.9999% of the times type-stable, so the macro anyways wouldn’t block anything, right?

Or are you worried for drastic increases in compile time?

I guess I just assumed that if your top-level functions were type stable, then your closures are also very likely type stable. So individual functions are a good “atomic unit” for debugging things.

And if only the closures themselves are unstable, but not the function, then the closures probably do something that isn’t important, like log debugging information or something (which wouldn’t need to be type stable anyways).

But if people want I could change that. I just thought it didn’t make much sense (but I welcome arguments otherwise, I just can’t think of any!).

1 Like

Could you implement a switch based on Preferences.jl? This way I could setup a CI test environment that has this enabled. Otherwise, it might be a good idea to disable this by default when deployed to the user.

4 Likes

This is incredibly useful.

It would indeed be nice to find closures. They often result in type instabilities that present a bunch of issues (see here for a longer description: https://github.com/tpapp/LogDensityProblemsAD.jl/pull/29#discussion_r1597685998).

Fun fact the fact that Threads.@threads creates a closure with this issue is why Enzyme actually vendors its own closure-free parallel for construct: Enzyme.jl/test/threads.jl at 1e45f264dbd2dacd79686c891f2d8c42ead33fce · EnzymeAD/Enzyme.jl · GitHub

3 Likes

Sure! Could you say a bit more about what you envision as the ideal setup?

Do you mean that all @stable in all packages could be enabled/disabled with a single Preferences.jl setting (sort of like it’s a compiler option), overriding any choice of enable?

Or would the enable option default to something set by Preferences.jl (again, like a compiler option), but a package could individually override it if they set it explicitly?

I haven’t used Preferences.jl before so not super clear on how it works.

Thanks!

Just to check, would the type instability still show up in the return value of the wrapping function? Or is it rather an internal type instability?

I think it could propagate through the return, but also may also just show up internally (depending on the closure definition).

Thanks. So if the instability propagates through the wrapping function’s return value then it should still get flagged (like this one). I guess the question then is if you would want to also have closures get flagged automatically if they don’t propagate to the return value.

The current behavior is to not do this (for the reasons here) but I could always change it, or maybe add another macro option to always recurse through functions.

I should also note that it still won’t be able to flag the closures generated by Threads.@threads unless those instabilities affect the function’s return value (since @stable would presumably do its processing before @threads). But maybe we could add special behavior for certain macros to support a lazy expansion.

If people start using this then great! Except, if in packages, then it can be type stable for e.g. types you care about (I guess all the main types), but then not for some exotic one, say for types like in Dec64.jl or for posits, when composing with packages that use @stable.

I didn’t check, but it would be nice to have an escape hatch, until people submit PRs to packages to help with type-stability. Likely an ENV var that turns your package off, or at least changes to warnings.

I think your package isn’t a prototype by now, should just work, would you ideally say it should be part of Base…? At least document its existence in Julia docs?

In Julia 2.0 possibly type-stable could be the default, though type-instability has its uses, then a/your macro to allow it. Possibly the new default should be at package/module level, i.e. for only certain packages (a certain 2.0 or new enough package, or a setting in Project.toml) and/or in debug/REPL mode?

Even better then checking for type-instability would be fixing it automatically… E.g. 0.0 → zero(x) and 1.0 to one(x), it’s just hard to know what that x should be… maybe it isn’t in some cases?

Preferences.jl uses your environment stack. So you can apply settings depot-wide or have them apply to a specific project. You can have this setting be in a LocalPreferences.toml or your Project.toml.

const enable_default = @load_preference("enable", false)

I suppose the question then is what should the default be. I contend the default should be false so when deployed to the user, who has no Preferences.jl setting, then the checks will be disabled. Meanwhile, the developer could have a LocalPreferences.toml in the package environment or in a test environment where this is enabled.

Just a quibble. Is the phrase “which is not a concrete type” in the error message applying to all cases ?

If that’s the case, it could be problematic. For instance, Vector{Any} is a concrete type.

Thanks. So I guess dispatch_doctor_warnonly and dispatch_doctor_enable make sense? Then you can set it for specific packages in your dependencies? (Is that how it works?)

I think we can do both:

  • If someone is using DispatchDoctor locally, like in a script, it would be nicer to have @stable work out of the box.
    • (I could see @stable being useful to Julia novices learning about performance gotchas – I wouldn’t want them to worry about LocalPreferences.toml yet)
  • If someone uses DispatchDoctor in a library, then indeed it would be nicer to disable it by default. And because they are a package dev, they are also more comfortable with customizing macro options/Preferences, so we can put that on them!

To deal with both cases, I think the enable option in the macro should be treated as a suggestion. If you wrap your package like

module MyPackage
using DispatchDoctor
@stable enable=false begin

# Entire package code

end
end

then the stability checks will be disabled by default. The dev could turn them on during testing with dispatch_doctor_enable => true in the preferences, but otherwise they would be set to off. So downstream users wouldn’t need to worry about them (unless an advanced user chooses to set them via LocalPreferences.toml)

In code, it would look like:

should_enable = load_preference(
    calling_module,  # __module__
    "dispatch_doctor_enable",
    enable           # Initial option passed to macro
)

which would produce the final result for whether it should be turned on or off.

Thoughts?

Not sure I follow? It seems to work fine:

julia> Base.isconcretetype(Vector{Any})
true

julia> f() = Any[1, 2]; Test.@inferred f()
2-element Vector{Any}:
 1
 2

julia> @stable g() = Any[1, 2]; g()
2-element Vector{Any}:
 1
 2

Okay made a PR here: Add preferences interface by MilesCranmer · Pull Request #14 · MilesCranmer/DispatchDoctor.jl · GitHub

You might find it useful to only enable @stable during unit-testing,
to have it check every function in a library, but not throw errors for
downstream users. For this, you can use the default_mode keyword to set the
default behavior:

module MyPackage
using DispatchDoctor
@stable default_mode="disable" begin

# Entire package code

end
end

This sets the default, but the mode is configurable
via Preferences.jl:

using MyPackage
using Preferences

set_preferences!(MyPackage, "instability_check" => "error")

which you can also set to be "warn" if you prefer.

1 Like

Very cool package!

I have not followed the performance optimization thread, so could you explain in terms accessible to a metaprogramming layperson how this is even possible? And whether we can be 100% certain that the check compiles away every time?

1 Like

So, because the Julia compiler infers return type of a function given the input types. All DispatchDoctor does is, for a function f1:

  1. Puts f1’s body into a separate function definition, with an identical signature, but a different name. Let’s call it f2.
  2. From the original function f1, it removes the original body, and inserts code that will checks the compiler-inferred type of f2 given the types of args.
  3. f1 runs Base.isconcretetype on that inferred type, and if false, throws an error.
  4. Otherwise, f1 calls f2(args…), and returns the result.

So because the inference of f2 and isconcretetype are in “type space”, it can do that at compile time. Thus, if things are type stable, Julia should just remove the error message from the code altogether.

Basically this if-statement’s first term only depends on input types:

And if type_instability(T) evaluates to false, it should compile to

if false && ...
    error()
end

f2(args...)

and then we assume LLVM will remove that if statement.

Well, I don’t want to say 100%. I haven’t found an example otherwise but with the Julia compiler and LLVM I never really feel 100% confident about an optimization taking place or not. It “should” compile away that if statement, but people should share examples otherwise.

I guess we could make ourselves 100% confident if we turn the checker function into a @generated function, so that we can explicitly remove the if statement from the code. But I’m assuming the Julia compiler will do that for us.

4 Likes

Thanks for the explanation! This is going straight into ModernJuliaWorkflows

10 Likes

Awesome :slight_smile:


By the way, does anybody have a workaround for using @stable directly on the top-level block of a Julia module?

I would like to do:

module MyPackage
@stable begin  # Wrap entire package
  
# pkg code...

# with some reexports:
using Reexport: @reexport
@reexport using MySubmodule: A, B, C
  
end
end

However, for this, it says @reexport is undefined. I looked more into this and it seems macros aren’t allowed to have their definition in a regular block:

begin
    macro m(ex); ex; end
    @m 1
end

This gives the same error. (e.g., Macros not imported within blocks)

So, to this, my question is: how can I apply @stable to the top-level block of a module, from within the module itself?

Basically I want for

@stable begin

end

to get “merged” into the same block it sits in, rather than creating a new block.