# Towards the creation of an interface testing package

**URL:** <https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182>\
**Category:** General Usage\
**Created:** [June 1, 2022, 10:25am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182 "2022-06-01T10:25:31Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [June 1, 2022, 10:25am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/1 "2022-06-01T10:25:31Z")

</div>

To get something actionable out of [this thread](https://discourse.julialang.org/t/discussion-on-why-i-no-longer-recommend-julia-by-yuri-vishnevsky/81151/), I keep thinking about an interface testing package.  
I also just found a glaring bug in DimensionalData.jl with mixed Int/CartesianIndex indexing. Because I just forgot to write a test for that.

Why are we all writing tests for our arrays and iterables when we are not all 100% across the interface? Why don’t we have standard testing facilities to make sure our implementations work?

On the flip side, for packages that consume these objects like the stats packages with bugs highlighted by the post: why don’t we have a package that builds the top 50 kinds of arrays or iterables from the ecosystem, that we can test against.

I noticed @oxinabox started this years ago:

> **[GitHub - oxinabox/InterfaceTesting.jl: (Julia 0.5 only) Concept still good,...](https://github.com/oxinabox/InterfaceTesting.jl)**
>
> (Julia 0.5 only) Concept still good, but needs updating for julia 1.0 - GitHub - oxinabox/InterfaceTesting.jl: (Julia 0.5 only) Concept still good, but needs updating for julia 1.0

Maybe something like that could be started up again?

---

<div class="post-metadata">

**Author:** ![holylorenzo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/holylorenzo/32/34169_2.png) [@holylorenzo](https://discourse.julialang.org/u/holylorenzo)\
**Post date:** [June 1, 2022, 10:50am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/2 "2022-06-01T10:50:35Z")

</div>

I am also working on a package that allows defining and composing invariants (incl. tests for interfaces) for

- providing helpful, precise error messages to package users when they misuse an API
- creating interface test suites

With a focus on

- reusability: invariants are easy to define and reuse, reducing boilerplate
- composability: invariants can be composed to create more complex invariants
- rich error messages: to be helpful, rich error messages should be easy to create

I am [using it in FastAI.jl](https://github.com/FluxML/FastAI.jl/pull/230) to give helpful errors to users and allow developers to test their implementations. For example, a lot of places make assumptions about the data passed in and the invariants give you precise information about which assumptions are met:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/5/853e4e88e0c2dcf16d0d62287744b90e61ba4520.jpeg)

For developers, you can start implementing an interface by defining a type, checking it against the invariant and just repeatedly fixing the errors it shows you until it’s all green.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [June 1, 2022, 11:16am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/3 "2022-06-01T11:16:20Z")

</div>

[Invariants.jl](https://github.com/lorenzoh/Invariants.jl) looks like a nice way to build modular interface tests!

I was wondering how to handle the problem that some objects only implement parts of the `AbstractArray` interface - there are a lot of optional pieces. But using Invariants.jl we could build up a tree of invariants and implementations can choose which branches to test.

It would also be nice to generate some automated interface documentation from this: like a list of what parts of the `AbstractArray` interface a package implements, that could go in the documentation somehow.

Maybe we could define a trait like:

```julia
implements(obj::MyArray) = (getindex_inv, setindex_inv, array_methods_inv)

```

Like some kind of poor-mans typeclasses. Then this could be used both in automated testing of invariants and to generate accurate documentation. It would be good to hear what ArrayInterface.jl people think.

---

<div class="post-metadata">

**Author:** ![holylorenzo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/holylorenzo/32/34169_2.png) [@holylorenzo](https://discourse.julialang.org/u/holylorenzo)\
**Post date:** [June 1, 2022, 11:35am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/4 "2022-06-01T11:35:26Z")

</div>

Yes, handling optional parts of an interface is doable here, since the interface test is composed out of smaller invariants. I’ll probably be doing this via kwargs to a function that builds the invariant, e.g. `check(my_interface(optional_thing_1=true, optional_thing_2=false), my_obj)`.

Love the idea of including the results of the automated checks in the docs!

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [June 1, 2022, 11:44am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/5 "2022-06-01T11:44:53Z")

</div>

Yeah lets try and get as much as possible out of defining this list! Maybe to make it available for other uses, instead of calling check like that, you could call it on the object then call the `implements` trait:

```julia
implements(::Type{<:MyObject}) = my_interface(optional_thing_1=true, optional_thing_2=false)

```

then in tests do:

```julia
check(obj)

```

and in docs like this would generate the markdown list of implemented traits and sub-traits:

```julia
@implements MyObject

```

and then in Intvariants define `check` like:

```julia
check(obj) = check(implements(obj), obj))

```

etc.

---

<div class="post-metadata">

**Author:** ![holylorenzo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/holylorenzo/32/34169_2.png) [@holylorenzo](https://discourse.julialang.org/u/holylorenzo)\
**Post date:** [June 1, 2022, 1:18pm UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/6 "2022-06-01T13:18:50Z")

</div>

I split this off into [an issue in Invariants.jl](https://github.com/lorenzoh/Invariants.jl/issues/2)

---

<div class="post-metadata">

**Author:** ![sbuercklin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sbuercklin/32/15728_2.png) [@sbuercklin](https://discourse.julialang.org/u/sbuercklin)\
**Post date:** [June 3, 2022, 1:48am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/7 "2022-06-03T01:48:50Z")

</div>

BinaryTraits.jl lets you define the proper methods necessary to support a trait and then test that the methods exist. If I understand correctly, this is similar to what you’re describing.

For documenting the interfaces/traits, I add docstrings to the trait types and then populate a page with the relevant trait type docstrings. With a proper abstract type hierarchy I would expect you could automate the documentation pages for interfaces instead of doing it manually.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [June 3, 2022, 2:35am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/8 "2022-06-03T02:35:26Z")

</div>

Isn’t a battery of tests that an implementation needs to pass equivalent to a definition of an interface? That is, a complete interface, which might not be desirable.

But assuming that is ok, separate out the tests that an interface must pass, and everyone can test against it.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [June 3, 2022, 6:21am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/9 "2022-06-03T06:21:19Z")

</div>

I think part of the problem is that it’s easy enough to forget to test against certain invariants, forget to update tests, etc. I think it would be nice from a usability standpoint if it was impossible to declare that a type implements an interface without at least having the correct methods with specific type signatures defined. I’m not sure if such a system can be built in Julia though because methods can be redefined for example, and whatever mechanism ensures interface compatibility would have to immediately say “no” if the interface isn’t fulfilled anymore.

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [June 3, 2022, 7:51am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/10 "2022-06-03T07:51:17Z")

</div>

Well, it’s possible to check that an interface is implemented correctly in any given world. Doing that automatically at call-time or something like that seems _really_ hard to make fast, but adding a `@test Base.satisfies_all_interfaces(MyType)` to a package’s tests seems very feasible. Ideally this info would also be exposed via a linter, which makes it significantly easier to write correct code (well, code that implements the right methods), but that is a bit trickier, of course.

---

<div class="post-metadata">

**Author:** ![nandoconde](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nandoconde/32/19497_2.png) [@nandoconde](https://discourse.julialang.org/u/nandoconde)\
**Post date:** [June 3, 2022, 9:29am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/11 "2022-06-03T09:29:05Z")

</div>

I can imagine, if not linter integration, at least a code analyzer _a là_ JET.jl

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [June 3, 2022, 9:42am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/12 "2022-06-03T09:42:23Z")

</div>

Is there a standard way to define interfaces in Julia?

I think there are a number of packages for that, like  
SimpleTraits.jl, WhereTraits.jl, BinaryTraits.jl etc…

I think before we can systematically test that interfaces are  
implemented correctly we need a standard way to define them.

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [June 3, 2022, 10:30am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/13 "2022-06-03T10:30:54Z")

</div>

There is not. I was just commenting on the general feasibility of testing whether an implementation conforms to a given interface, however that is going to be defined.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [June 3, 2022, 10:54am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/14 "2022-06-03T10:54:06Z")

</div>

Could that be done in the time-frame of Julia 1.x, or would that have to wait for Julia 2.x ?

---

<div class="post-metadata">

**Author:** ![pablosanjose](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pablosanjose/32/7006_2.png) [@pablosanjose](https://discourse.julialang.org/u/pablosanjose)\
**Post date:** [June 3, 2022, 12:16pm UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/15 "2022-06-03T12:16:24Z")

</div>

For reference: [Interfaces for Abstract Types · Issue #6975 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/6975)

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [June 3, 2022, 12:17pm UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/16 "2022-06-03T12:17:10Z")

</div>

We just need to write a package, it diesnt have to be in base.

Im going to experiment with an Interfaces.jl implementation at some stage, it should be quite simple, like 100 lines. The hard work will be getting Base interfaces ike AbstractArray implemented and correct.

@pfizeb we dont need to check the interface it at call time, we can just set up the traits and check them in `precompile`. Which has multiple benefits.

---

<div class="post-metadata">

**Author:** ![TheCedarPrince](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thecedarprince/32/17323_2.png) [@TheCedarPrince](https://discourse.julialang.org/u/TheCedarPrince)\
**Post date:** [July 1, 2022, 12:10am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/17 "2022-07-01T00:10:57Z")

</div>

Hey @Raf ,

I am very curious about addressing this notion of interfaces.  
I am beginning to create some packages for observational health and health equity research; I certainly want to maintain a high level of correctness there.  
I tried to search your GitHub but did not see an interfaces package anywhere.

Admittedly, I do not know too much about correct and verifiable interface design but I am willing to learn and to try.  
If you want to set-up a call to do a bit of a coding session and planning discussion to put together an initial draft, happy to do so.

Thanks!

~ tcp 🌳

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [July 1, 2022, 9:27am UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/18 "2022-07-01T09:27:58Z")

</div>

Ok just pushed [GitHub - rafaqz/Interfaces.jl: Macros to define and implement interfaces, to ensure they are checked and correct.](https://github.com/rafaqz/Interfaces.jl) . Its just a bunch of macro definitions written one evening that dont actually work yet. And some examples for Base.

But the idea is to have a `@defines` macro that defines interface tests and traits and an `@implements` macro that connects an object to the interface with the ability to split it into components. E.g `@implements MyObject IteratorInterface{(:reverse,:indexing)}`.

It would define a trait for which components are implemented, link the object to the interface tests and documentation, and optionally run the tests during precompilation.

---

<div class="post-metadata">

**Author:** ![TheCedarPrince](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thecedarprince/32/17323_2.png) [@TheCedarPrince](https://discourse.julialang.org/u/TheCedarPrince)\
**Post date:** [July 1, 2022, 4:40pm UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/19 "2022-07-01T16:40:36Z")

</div>

This is great!  
As I am quite new to this, what would be useful to try out on my part?

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [July 1, 2022, 4:43pm UTC](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182/20 "2022-07-01T16:43:04Z")

</div>

I mean, nothing works yet!

But just reading it and giving feedback would be great. Mostly I’m just trying to think about what could work and get the most out of making these declarations. The code will be pretty simple.

It’s probably not even ready to read really, why I hadn’t shared it already. But maybe I can clean it up over the weekend. Or now… just realised I hadn’t included the Base interface examples for arrays and iteration, theyre pushed now.

[Next page](https://discourse.julialang.org/t/towards-the-creation-of-an-interface-testing-package/82182.md?page=2)