To get something actionable out of this thread, I keep thinking about an interface testing package.
I also just found a glaring bug in DimensionalData.jl with mixed Int/CartesianIndex indexing. Because I just forgot to write a test for that.
Why are we all writing tests for our arrays and iterables when we are not all 100% across the interface? Why don’t we have standard testing facilities to make sure our implementations work?
On the flip side, for packages that consume these objects like the stats packages with bugs highlighted by the post: why don’t we have a package that builds the top 50 kinds of arrays or iterables from the ecosystem, that we can test against.
I am also working on a package that allows defining and composing invariants (incl. tests for interfaces) for
providing helpful, precise error messages to package users when they misuse an API
creating interface test suites
With a focus on
reusability: invariants are easy to define and reuse, reducing boilerplate
composability: invariants can be composed to create more complex invariants
rich error messages: to be helpful, rich error messages should be easy to create
I am using it in FastAI.jl to give helpful errors to users and allow developers to test their implementations. For example, a lot of places make assumptions about the data passed in and the invariants give you precise information about which assumptions are met:
For developers, you can start implementing an interface by defining a type, checking it against the invariant and just repeatedly fixing the errors it shows you until it’s all green.
Invariants.jl looks like a nice way to build modular interface tests!
I was wondering how to handle the problem that some objects only implement parts of the AbstractArray interface - there are a lot of optional pieces. But using Invariants.jl we could build up a tree of invariants and implementations can choose which branches to test.
It would also be nice to generate some automated interface documentation from this: like a list of what parts of the AbstractArray interface a package implements, that could go in the documentation somehow.
Like some kind of poor-mans typeclasses. Then this could be used both in automated testing of invariants and to generate accurate documentation. It would be good to hear what ArrayInterface.jl people think.
Yes, handling optional parts of an interface is doable here, since the interface test is composed out of smaller invariants. I’ll probably be doing this via kwargs to a function that builds the invariant, e.g. check(my_interface(optional_thing_1=true, optional_thing_2=false), my_obj).
Love the idea of including the results of the automated checks in the docs!
Yeah lets try and get as much as possible out of defining this list! Maybe to make it available for other uses, instead of calling check like that, you could call it on the object then call the implements trait:
BinaryTraits.jl lets you define the proper methods necessary to support a trait and then test that the methods exist. If I understand correctly, this is similar to what you’re describing.
For documenting the interfaces/traits, I add docstrings to the trait types and then populate a page with the relevant trait type docstrings. With a proper abstract type hierarchy I would expect you could automate the documentation pages for interfaces instead of doing it manually.
Isn’t a battery of tests that an implementation needs to pass equivalent to a definition of an interface? That is, a complete interface, which might not be desirable.
But assuming that is ok, separate out the tests that an interface must pass, and everyone can test against it.
I think part of the problem is that it’s easy enough to forget to test against certain invariants, forget to update tests, etc. I think it would be nice from a usability standpoint if it was impossible to declare that a type implements an interface without at least having the correct methods with specific type signatures defined. I’m not sure if such a system can be built in Julia though because methods can be redefined for example, and whatever mechanism ensures interface compatibility would have to immediately say “no” if the interface isn’t fulfilled anymore.
Well, it’s possible to check that an interface is implemented correctly in any given world. Doing that automatically at call-time or something like that seems really hard to make fast, but adding a @test Base.satisfies_all_interfaces(MyType) to a package’s tests seems very feasible. Ideally this info would also be exposed via a linter, which makes it significantly easier to write correct code (well, code that implements the right methods), but that is a bit trickier, of course.
There is not. I was just commenting on the general feasibility of testing whether an implementation conforms to a given interface, however that is going to be defined.
We just need to write a package, it diesnt have to be in base.
Im going to experiment with an Interfaces.jl implementation at some stage, it should be quite simple, like 100 lines. The hard work will be getting Base interfaces ike AbstractArray implemented and correct.
@pfizeb we dont need to check the interface it at call time, we can just set up the traits and check them in precompile. Which has multiple benefits.
I am very curious about addressing this notion of interfaces.
I am beginning to create some packages for observational health and health equity research; I certainly want to maintain a high level of correctness there.
I tried to search your GitHub but did not see an interfaces package anywhere.
Admittedly, I do not know too much about correct and verifiable interface design but I am willing to learn and to try.
If you want to set-up a call to do a bit of a coding session and planning discussion to put together an initial draft, happy to do so.
But the idea is to have a @defines macro that defines interface tests and traits and an @implements macro that connects an object to the interface with the ability to split it into components. E.g @implements MyObject IteratorInterface{(:reverse,:indexing)}.
It would define a trait for which components are implemented, link the object to the interface tests and documentation, and optionally run the tests during precompilation.
But just reading it and giving feedback would be great. Mostly I’m just trying to think about what could work and get the most out of making these declarations. The code will be pretty simple.
It’s probably not even ready to read really, why I hadn’t shared it already. But maybe I can clean it up over the weekend. Or now… just realised I hadn’t included the Base interface examples for arrays and iteration, theyre pushed now.
