I don't get Base.IteratorElType

Internals & Design
, , , ,
1

Motivation

I’m implementing a collect-like function for a certain subtype of AbstractArray, FixedSizeArray (@giordano). The function is supposed to take an arbitrary iterator, consume it, and return a FixedSizeArray value constructed from the iterator. While trying to do this, however, I realized I don’t understand anything about Base.IteratorElType.

Base.IteratorElType

What goals is Base.IteratorElType supposed to accomplish? What does it solve? What are its semantics in practice?

Empirically, the only place where I’ve found that Base.IteratorElType matters is for collect:

julia> it = Ref{Any}(7)
Base.RefValue{Any}(7)

julia> Base.IteratorEltype(it)
Base.HasEltype()

julia> collect(it)
0-dimensional Array{Any, 0}:
7

julia> map(identity, it)
0-dimensional Array{Int64, 0}:
7

julia> typeof(it)
Base.RefValue{Any}

julia> Base.IteratorEltype(::Type{Base.RefValue{Any}}) = Base.EltypeUnknown()

julia> Base.IteratorEltype(it)
Base.EltypeUnknown()

julia> collect(it)
0-dimensional Array{Int64, 0}:
7

julia> map(identity, it)
0-dimensional Array{Int64, 0}:
7

So Base.IteratorElType affects the behavior of collect, while it does not affect map. The effect is that setting the return value to Base.EltypeUnknown() makes collect behave more like map, otherwise it just preserves the element type exactly.

Is this really the only place where Base.IteratorElType matters, for collect? Is the current behavior even desirable? I guess returning Array{Int} would actually be nicer than returning Array{Any}?

Questions regarding the design of the new function

Basically I wonder if bothering with Base.IteratorElType while implementing the new function is even necessary.

This is the interface I imagine so far:

"""
    collect_as(t::Type{<:FixedSizeArray}, iterator)

Tries to construct a value of type `t` from the iterator `iterator`. The type `t`
must either be concrete, or a `UnionAll` without constraints.
"""
function collect_as(::Type{T}, iterator) where {T<:FixedSizeArray}
    ...
end

My intention is basically to have collect_as behave like so:

  1. If the element type is given, like, for example, in collect_as(FixedSizeArray{Int}, iter), where the element type is given as Int, the returned value must have the requested element type. So Base.IteratorElType doesn’t need to come into the picture.

  2. If no element type is requested, like, for example, in collect_as(FixedSizeArray, iter), the intention is to ignore Base.IteratorElType, store the elements of the iterator into a temporary FixedSizeArray value whose element type is eltype(iter), then, at the end, return map(identity, temporary_fixed_size_array), so as to get a tight element type. So I’m ignoring Base.IteratorElType, is that OK?

1 Like
2

Hi!

I personally don’t think it is a good idea. Looks like Base.EltypeUnknown() means that the eltype function is not even defined for the iterator (or, possibly, not expected to return sensible results).

You can use @less collect(it) to peek into the source code of the collect function method that gets called to see the difference in these two cases.

3

I thought so too, at first, however I don’t think that makes sense from a design standpoint. The default definition of eltype returns Any, which is always correct, so someone defining an iterator type would have to go out of their way to make eltype throw or return an incorrect value.

Furthermore, if that’s the only issue, I could simply fix my algorithm by using something like eltype_fixed, given below, instead of using eltype directly:

eltype_fixed_impl(::Type, ::Base.EltypeUnknown) = Any
eltype_fixed_impl(::Type{T}, ::Base.HasEltype) = eltype(T)
eltype_fixed(::Type{T}) where {T} = eltype_fixed_impl(T, Base.IteratorElType(T))
4

OK, I conducted some experiments. EltypeUnknown triggers usage of the Base.@default_eltype macro which infers the eltype of the iterator (accepts an iterable object, not its type!) using some compiler trickery I didn’t fully understand (source here).

Also eltype returns Any by default for generators, so comprehensions will yield Any-typed arrays. This may be fixed by defining eltype_fixed for objects and not types… But, apparently, there is a reason why eltype does not call Base.@default_eltype by default.

5

Is this FixedSizeArray type meant to be a PR for Base? If it’s going into a package, I’m not sure that referencing how collect and map use or don’t use Base.IteratorEltype is a good idea, since various code paths for collect and map rely on magical things like Base.@default_eltype that are not available to package authors.

Other than that, I can’t provide you much advice on how to use Base.IteratorEltype other than what’s in the docstring (which I’m sure you’ve seen):

This trait is generally used to select between algorithms that pre-allocate a specific type of result, and algorithms that pick a result type based on
the types of yielded values.

1 Like
6

See also this PR for

into(T::Type, iterable) -> collection::T
1 Like
7

FWIW this is the PR: