Could introducing formal interfaces be nonbreaking?

Internals & Design
21

At any rate, I’m doubtful that runtime interface checking (e.g., via the JIT compiler) would be a good fit for Julia. Interface checking could make sense in a static analysis context, like JET.jl, and it definitely makes sense to have a unit test suite for interfaces.

For example, consider the original example of an interface specification provided by Chris:

f(a::T, b::T)::T

This form of interface specification is quite limiting in the current Julia world of single inheritance. In particular, I wouldn’t be able to declare a function signature like this:

f(a::Iterator, b::Iterator)::Iterator

(Though I don’t know, maybe the compiler folks have ideas for more complicated interface specifications?)

Julia has never had the ability or semantics to constrain the output type of generic functions. It seems counter to the dynamic nature of the language to add it now, and I suspect it would lead to unintended consequences.

1 Like
22

A breaking change is about the API and has never meant anything that made any code stop working. If that were the case, changing internals would be breaking changes because it disrupts packages that rely on them. For obvious reasons, internals don’t print warnings or throw errors every time they’re used. Docstrings may very well need correction sometimes, but that doesn’t mean a random user’s unilateral decision to defy the documented API becomes API.

It’s worth pointing out that even formal interfaces in general may not throw all the warnings and errors we need; static type analysis will run into a wall every time there’s a runtime dispatch, even if the only existing methods do conform to the interface. Sometimes it may just be like public; something we can look up interactively without having to search text in the documentation and otherwise completely silent.

1 Like
23

nobody is saying the criterial is “any code”, people are specifically talking about things like ==() which is a public function, and it so far does not required to always return Bool, or take the example of AbstractArray, there’s not a set of interface package that subtype this must satisfy. So if one day someone’s package which defines T <: AbstractArray suddenly starts to break, you’ve introduced a breaking change, which is not allowed in Julia 1.x

24

Really, because I also see this there now:

This seems pretty clear: Interfaces: AbstractArray

Again, this is just the colloquial sense of code failing. Breaking changes are entirely about what the developer does to their API. A breaking change could by chance not affect anyone’s code, trivially if nobody is using it.

25

Is missing a DSL? Because missing == missing -> missing which is not a Bool. Why is missing special? Should interfaces allow for specifically having a different rule on one concrete type, or should the rule have to apply to all types?

I would argue that if we have to single out specific types in the rule for which the rule doesn’t apply to, we’ve got the rule wrong and should go back to the drawing board. We’ve always said Base isn’t special, we could do things in a package, but here is a case where MyMissings.jl would not be possible by definition. Are we okay with losing that fundamental tenant of Julia for a simpler interface on this function?

It’s not entirely, it’s actually quite ambiguous.

size(A): Returns a tuple containing the dimensions of A

Are those dimensions supposed to be scalar? Or for example with a ragged array, should it be a tuple of vectors for the dimensions of each vector?

Base code generally assumes that size(A) is a tuple of scalars, but it’s not actually in the interface definition.

getindex(A, i::Int): (if IndexLinear) Linear scalar indexing

Should A[i]::T where eltype(A)==T? That isn’t required in the definition of the interface, but much of Base code relies on the idea that scalar indexing always gives something of eltype(A). A classic type which violated this was always Tracker.jl’s TrackedArray, which stated eltype(A) == Float64 but a[i] == TrackedReal{Float64}.

And for that matter, is eltype(::AbstractArray{T,N}) where {T,N} = T supposed to be true? Because the interface does not specify how eltype works.

getindex(A, I::Vararg{Int, N}): (if IndexCartesian, where N = ndims(A)) N-dimensional scalar indexing

Is the identity getindex(A, Base.Cartesian.CartesianIndex(i)) == getindex(A, i) expected to hold? Because that’s not specified.

setindex!(A, v, i::Int): (if IndexLinear ) Scalar indexed assignment


julia> a = rand(2)
2-element Vector{Float64}:
 0.8135346986182869
 0.7300081330807777

julia> a[1]= 2
2

Is it expected that setindex!(A, v::T, i::Int)::T, i.e. that it also returns v? Again there is some Base code in the AbstractArray that relies on this, but it’s not necessarily stated as something that should be true for all AbstractArrays

length(A): Default definition prod(size(A))

Is this expected to be true, or just a default? There are some codes (IIRC broadcast is one?) that assumes this is true.

similar(A): Return a mutable array with the same shape and element type

Is CuArray considered mutable if allowscalar is false? Because then A[1] = v always errors. How do we define “mutable” here: that A[i] = v is a valid non-erroring operation? Since if it’s the latter, similar(A::CuArray)::CuArray is not correct.

Other functions aren’t even mentioned. Is zero(x::T)::T an identity we expect for array types? The only case I know that violates that is Measurements.jl.

I highly encourage folks to think beyond their own immediate codes when thinking about interface definitions and think holistically about how it applies to other codes and the downstream effects. When doing so, you get some not necessarily an easy questions to answer, and its okay for us to document the current limitations of our knowledge and understand that we commonly get interface definitions wrong or ambiguous.

9 Likes
26

I ask myself this question frequently… the fact that == with missing infects collections that could possibly potentially maybe contain a missing is so annoying. or the fact that it takes any and all hostage with mandatory three-valued logic.

6 Likes
27

I stand corrected. We wouldn’t even need to wait for much formality, it’d be good just to clarify which of those assumptions were actually intended to be part of the interface. And that’s a tougher collaborative job because it’s worth weighing whether the interface can include the challenging implementations.

Seems like setindex! returns A, though that’s an assumption worth arguing over:

julia> setindex!(a, 3, 2)
2-element Vector{Float64}:
 2.0
 3.0

It’s only the assignment syntax that evaluates to the right hand expression, and that guarantee isn’t part of the AbstractArray interface.

2 Likes
28

As said, there’s many potential options, but the most obvious option would be an interface where you can insert checks into your test suite to make sure you’ve satisfied it.

E.g. staying with the == example, if Base did

@interface ((==)(::T, ::U) where {T, U})::Bool

then say you wrote

struct Foo end
Base.:(==)(f::Foo, x) = f

then we could make it so that this doesn’t throw any errors, but then in your test suite you could have

@test satisfies_interface((==), Foo())

or whatever, and have that be responsible for throwing errors. We could even potentially integrate something like this into Aqua.jl so you’re alerted and automatically tested if you touch any interface methods.

Adding interfaces in this way would be non-breaking.

There’s multiple reasons we might want to go this way too, not just because it’d be non-breaking. Another reason to do it this way is that gasp not all interface contracts can be expressed or checked statically. Sometimes they depend on runtime values, so we’d presumably want to include actual value-based runtime checks in our interfaces, and we definitely wouldn’t want those checks to run each time you call a function which is part of an interface (unless you’re in some sort of special debug-mode)

7 Likes
29

Please add some links to make the discussion more concrete.

There have been many proposals and discussions.

30

The context here is that this thread was deemed off topic and split off from Efficiency of abstract types vs `Any`, where Chris’s comment mentioned “when interfaces are added to the language”

Chris didn’t really get into any particular interface design there, but the mention of it piqued some interest from people, including Mateus, and another thread which was separately split off.

1 Like
31

I’d be happier with something like strict mode. That gives a path for the language to evolve. I’m afraid that the current focus on avoiding breaking changes at any cost will lead to stagnation.

A test-based approach is less impactful, because it requires either a test to be written for every function, or for something like Aqua to be run in CI (which is almost never done). The default is to be lazy and not do it, and the inevitable result is to have countless of silent interface violations.

32

And indeed the hole was filled with rabbits :rabbit_face: :rabbit_face: :rabbit_face: :rabbit_face: :rabbit_face: :rabbit_face:

6 Likes
33

I understand now. This happened many times before, so from a practical perspective, I do not have to fear formal interfaces as being imminent :wink:

In this particular case, all is required is though is inferring the return type. That could be accomplished with a much more restricted, opt-in addition to the language where that is defined.

34

From the performance tips:

In the case that the type of a[1] is not known precisely, x can be declared via x = convert(Int32, a[1])::Int32. The use of the convert function allows a[1] to be any object convertible to an Int32 (such as UInt8), thus increasing the genericity of the code by loosening the type requirement. Notice that convert itself needs a type annotation in this context in order to achieve type stability. This is because the compiler cannot deduce the type of the return value of a function, even convert, unless the types of all the function’s arguments are known.

This is just begging for having formal interfaces. And it is not achievable in a non-breaking way, even in strict mode. Unless all your dependencies are also in strict mode.

36

Relevant issue:

2 Likes
37

I think you’re confusing with one (and Measurements.jl follows precisely the one/oneunit semantic, doesn’t violate anything):

julia> using Measurements

julia> x = measurement(1.0)
1.0 ± 0.0

julia> typeof(x)
Measurement{Float64}

julia> zero(x)
0.0 ± 0.0

julia> typeof(zero(x))
Measurement{Float64}
1 Like
38

Right, and similarly zero isn’t guaranteed to be the same type; it’s just the (or an?) additive identity. E.g.,

julia> using Dates

julia> typeof(now()), typeof(zero(now()))
(DateTime, Millisecond)
2 Likes