Is OffsetArrays.jl a poison pill?

Internals & Design
23

Ouch. Yes, I typed this from my phone, so dropped testing it :frowning:

24

Certainly there is no AbstractTuple at the moment but, given the flexibility of the language, there is no reason why AbstractTuples and OffsetTuples could not exist in the future. Should the use of these become widespread then this could cause disruption to the existing code base. Much of the discussion so far has concerned AbstractArrays but these sorts of problems could arise in the future with any of the abstract types. I guess this is one of the issues that Yuri was concerned about.

1 Like
25

No, this would require changes in Base itself (or even Core?), the type hierarchy with Any → Tuple is hardcoded, and you cannot wedge AbstractTuple in there from the outside. It is not something that is possible to do from a package, or somehow from ‘the outside’, so it’s not related to the flexibility of the language.

If you allow for changes to the core language, then of course anything could happen, but that’s obvious.

2 Likes
26

I have missed AbstractTuple as a way to let concrete Tuples, NTuples and NamedTuples more easily play in the same sandbox[s]. One advantage would be an easily seen uniform convention for sequential indexing of the ith element of a Tuple, or the ith value of a NamedTuple. Bringing NamedTuples closer in devthink to Tuples elaborated with position-attached names (symbols) would make working flexibly with them easier for many imo.

27 
julia> NTuple{2, Int} === Tuple{Int, Int}
true

NTuple is just an alias.

1 Like
28

As someone who was in the “1-based to rule them all” camp for a while during that discussion, I can totally see the appeal of separating OffsetArrays somehow. But I came to realize some things:

  1. There’s no decree to use or support OffsetArrays, and we can totally write 1:n syntax in that case. We even have the require_one_based_indexing function to check inputs. Funnily enough, if there’s a custom array type that isn’t 1-based, OffsetArrays can make it 1-based to be compatible with your 1-based code.

  2. OffsetArrays are actually important. You should see the cool examples on the Julia blog post introducing them, but it’s not hard to run into math where starting at 1 doesn’t make sense. We’d have to do offset calculations to get to the index we need anyway, why not make a wrapper type do it automatically?

  3. Why shouldn’t OffsetArrays work in AbstractArray code? Many functions really could work with any indices, and some functions currently marked with require_one_based_indexing actually could work with offset indices, just require that all inputs share compatible ones. Incompatible indices can just throw the existing DimensionMismatch error.

  4. We definitely should make a docs section or tutorial that teaches people right away how to write “generic” indices; people might not be as good as indexing arrays as they think. (Even the devs! The OffsetArray discussion revealed some overflow bugs when indices or related values get too close to typemax/typemin.) It’s not a big deal when you’re just indexing from begin to end (and in that case, just call eachindex(A) or axes(A, i)), but it gets real important when you start doing fancy stuff like indexing the “middle”, skipping every other index, or going backwards. Obscure errors happen when different key quantities that happen to have the same values are confused with each other, like an n meaning the last index vs. an n meaning length. In 0-based indexing, this is actually more annoying because there are invisible 0+ everywhere that you have to spot to edit the indexing order. begin/end syntax could be improved to make these distinctions easier, and even if you don’t do offset indices you’d reach for the generic syntax just for clarity.

9 Likes
29

Yes, Certainly. Additionally, I have found it a clearly readable way of saying
“All the elements present are of type T, and there are N of them”, the first part is obvious enough with Tuple{Int, Int, Int, Int, Int, Int, Int, Int}, the second less so.

30

But then you don’t need AbstracTuple to ‘let them play in the same sandbox’?

1 Like
31

Rather than doing this in a function in your normal code or in a standalone test in a testrunner I rahter see this check done by a linter/compiler via the type system.

In general that’s the evolution that you see in programming languages over the decades. In the beginning you have languages with pretty weak type systems (e.g. C) so if you wanted to enforce (complex) constraints you had to do it in runtime code. Later on type system got more sofisticated and a lot of checks could be offloaded to that (e.g. ML, Haskell, Scala, F#,…)

There’s a saying (usually for static vs dynamic languages but applies more widely): every type check is a bunch of tests you don’t have to write (or can forget to write).

32

about 4:
If you’re just going loop over all elements I find “for e in A … end” the better way than eachIndex and then indexing. It’s just nice syntactic sugar that abstracts all that stuff and on the plus side also column based vs row based matrices or even if it’s a vector, matrix or whatever iterable collection.

1 Like
33

In other words, the iteration interface. I do see people unnecessarily iterating over indices just to getindex one array only once, good to point out. Iteration isn’t good enough when you need setindex!, though.

34

“With the introduction of OffsetArrays.jl AbstractArrays now confounds abstraction of element data type with the abstraction of indexing”

Actually the abstraction wasn’t about element type (that’s covered by generic type parameters) but was more about underlying storage type like BLAS matrix, distributed array, off-heap (on-disk) array,…

I agree with the confounding now with indexing and trouble of (retro)fitting the type hierarchy. If you wanted to shove storage type & indexing scheme into a type hierarchy you’ll have a multiplier effect and hence a lot of leaf nodes. A better way is to go more general and go with a type graph: i.e. have a type implement multiple traits/interfaces (yes here are traits again!). So you would have a trait hierarchy for storage type and one for indexing and a type would pick one from each.

2 Likes
35

Yes, and also about structure or properties, like symmetry and diagonal, or normalization, etc.

36

Yes it can be about many aspects of Arrays! The point is it now it is all thrown together into one hierarchy and a single type has many aspects which are not always obvious from the type name. When all that stuff would be refactored into separate traits / trait hierarchies one could compose each aspect indepently and have the type checker also validate each aspect indepently. That’s really powerful stuff!

37

@jishnub

so it would be something like this?

for t in first(axes(x, 1))+3:last(axes(x, 1)), i in first(axes(x, 2)+1:last(axes(x, 2))
38

This could be

firstindex(x, 1)+3:lastindex(x, 1)
1 Like
39

yes, that’s a lot better. thanks

40

I think you’re painting things as excessively binary where you choose exactly one of correctness or composability. Having just those options is quite dangerous for Julia’s reputation because it will cause more people to think “Julia doesn’t care about correctness”.

I would focus on several dimensions where there’s more options:

  1. Which code should be composable?
  2. Which kinds of bounds on composition should exist?

On (1), I would encourage the community to not tell every new Julia programmer that composability is the highest virtue. Indeed, I think high composability is not the right default goal for user code and perhaps not the right goal for most packages either.

And (2) is also important: the style guide I quoted can thought of as saying that there should be no bounds on function arguments, but I suspect if pushed it would be refined to “only exclude types known to fail”. But it would be entirely reasonable to instead default to only allowing types known to work.

6 Likes
41

So, my intention was to ask if you saw this as a binary choice. Thanks for answering.

2 Likes
42

I think part of the problem is that currently the ecosystem is in a “frustated” state: on the one hand thanks to the flexibility of the language it’s possible to define all sorts of new array types that come with non-standard (at least compared to the old 1-based standard) indexing , but on the other hand the type hierarchy / type system hasn’t followed to really precisely express what types are supported by a package.

This then leads to the by now well discussed problems. Right now the ecosystem is sitting at an unstable point and it could go either way: descend into chaos or evolve the type system and move to a better place which is not only very generic for array types but also correct/safe.

1 Like