# I don't get Base.IteratorElType
**URL:** https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604
**Category:** Internals & Design
**Tags:** question, array, design, collection, iterators
**Created:** [April 29, 2024, 8:22am UTC](https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604 "2024-04-29T08:22:38Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@nsajko](https://discourse.julialang.org/u/nsajko)
#### Post date: [April 29, 2024, 8:22am UTC](https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604/1 "2024-04-29T08:22:38Z")
## Motivation
I’m implementing a `collect`-like function for a certain subtype of `AbstractArray`, `FixedSizeArray` (@giordano). The function is supposed to take an arbitrary iterator, consume it, and return a `FixedSizeArray` value constructed from the iterator. While trying to do this, however, I realized I don’t understand anything about `Base.IteratorElType`.
## `Base.IteratorElType`
What goals is `Base.IteratorElType` supposed to accomplish? What does it solve? What are its semantics in practice?
Empirically, the only place where I’ve found that `Base.IteratorElType` matters is for `collect`:
```julia-repl
julia> it = Ref{Any}(7)
Base.RefValue{Any}(7)
julia> Base.IteratorEltype(it)
Base.HasEltype()
julia> collect(it)
0-dimensional Array{Any, 0}:
7
julia> map(identity, it)
0-dimensional Array{Int64, 0}:
7
julia> typeof(it)
Base.RefValue{Any}
julia> Base.IteratorEltype(::Type{Base.RefValue{Any}}) = Base.EltypeUnknown()
julia> Base.IteratorEltype(it)
Base.EltypeUnknown()
julia> collect(it)
0-dimensional Array{Int64, 0}:
7
julia> map(identity, it)
0-dimensional Array{Int64, 0}:
7
```
So `Base.IteratorElType` affects the behavior of `collect`, while it does _not_ affect `map`. The effect is that setting the return value to `Base.EltypeUnknown()` makes `collect` behave more like `map`, otherwise it just preserves the element type exactly.
Is this really the only place where `Base.IteratorElType` matters, for `collect`? Is the current behavior even desirable? I guess returning `Array{Int}` would actually be nicer than returning `Array{Any}`?
## Questions regarding the design of the new function
Basically I wonder if bothering with `Base.IteratorElType` while implementing the new function is even necessary.
This is the interface I imagine so far:
```julia
"""
collect_as(t::Type{<:FixedSizeArray}, iterator)
Tries to construct a value of type `t` from the iterator `iterator`. The type `t`
must either be concrete, or a `UnionAll` without constraints.
"""
function collect_as(::Type{T}, iterator) where {T<:FixedSizeArray}
...
end
```
My intention is basically to have `collect_as` behave like so:
1. If the element type is given, like, for example, in `collect_as(FixedSizeArray{Int}, iter)`, where the element type is given as `Int`, the returned value must have the requested element type. So `Base.IteratorElType` doesn’t need to come into the picture.
2. If no element type is requested, like, for example, in `collect_as(FixedSizeArray, iter)`, the intention is to ignore `Base.IteratorElType`, store the elements of the iterator into a temporary `FixedSizeArray` value whose element type is `eltype(iter)`, then, at the end, return `map(identity, temporary_fixed_size_array)`, so as to get a tight element type. So I’m ignoring `Base.IteratorElType`, is that OK?
---
### Author:  [@aryavorskiy](https://discourse.julialang.org/u/aryavorskiy)
#### Post date: [April 29, 2024, 10:13am UTC](https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604/2 "2024-04-29T10:13:51Z")
Hi!
I personally don’t think it is a good idea. Looks like `Base.EltypeUnknown()` means that the `eltype` function is not even defined for the iterator (or, possibly, not expected to return sensible results).
You can use `@less collect(it)` to peek into the source code of the `collect` function method that gets called to see the difference in these two cases.
---
### Author:  [@nsajko](https://discourse.julialang.org/u/nsajko)
#### Post date: [April 29, 2024, 10:32am UTC](https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604/3 "2024-04-29T10:32:06Z")
> [@aryavorskiy](#):
>
> Looks like `Base.EltypeUnknown()` means that the `eltype` function is not even defined for the iterator (or, possibly, not expected to return sensible results).
I thought so too, at first, however I don’t think that makes sense from a design standpoint. The default definition of `eltype` returns `Any`, which is always correct, so someone defining an iterator type would have to go out of their way to make `eltype` throw or return an incorrect value.
Furthermore, if that’s the only issue, I could simply fix my algorithm by using something like `eltype_fixed`, given below, instead of using `eltype` directly:
```julia
eltype_fixed_impl(::Type, ::Base.EltypeUnknown) = Any
eltype_fixed_impl(::Type{T}, ::Base.HasEltype) = eltype(T)
eltype_fixed(::Type{T}) where {T} = eltype_fixed_impl(T, Base.IteratorElType(T))
```
---
### Author:  [@aryavorskiy](https://discourse.julialang.org/u/aryavorskiy)
#### Post date: [April 29, 2024, 11:07am UTC](https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604/4 "2024-04-29T11:07:00Z")
OK, I conducted some experiments. `EltypeUnknown` triggers usage of the `Base.@default_eltype` macro which infers the eltype of the iterator (accepts an iterable object, not its type!) using some compiler trickery I didn’t fully understand [(source here)](https://github.com/JuliaLang/julia/blob/master/base/array.jl#L740-L767).
Also `eltype` returns `Any` by default for generators, so comprehensions will yield `Any`-typed arrays. This may be fixed by defining `eltype_fixed` for _objects_ and not _types_… But, apparently, there is a reason why `eltype` does not call `Base.@default_eltype` by default.
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [April 30, 2024, 1:43am UTC](https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604/5 "2024-04-30T01:43:31Z")
Is this `FixedSizeArray` type meant to be a PR for Base? If it’s going into a package, I’m not sure that referencing how `collect` and `map` use or don’t use `Base.IteratorEltype` is a good idea, since various code paths for `collect` and `map` rely on magical things like `Base.@default_eltype` that are not available to package authors.
Other than that, I can’t provide you much advice on how to use `Base.IteratorEltype` other than what’s in the docstring (which I’m sure you’ve seen):
> This trait is generally used to select between algorithms that pre-allocate a specific type of result, and algorithms that pick a result type based on
> the types of yielded values.
---
### Author:  [@jar1](https://discourse.julialang.org/u/jar1)
#### Post date: [April 30, 2024, 1:46am UTC](https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604/6 "2024-04-30T01:46:19Z")
See also this PR for
```julia
into(T::Type, iterable) -> collection::T
```
>
>
> close #36288
>
> This PR implements the API I proposed in #36288. Here is the (re…fined) API I ended up implementing in this PR. Quoting the docstrings:
>
> \# Surface API
>
> \`\`\`julia
> into(T::Type, iterable) -\> collection::T
> into(T::Type) -\> iterable -\> collection::T
> \`\`\`
>
> Construct a new \`collection\` of type \`T\` that contains the elements in \`iterable\`. If \`iterable\` is also a container, it acts as a shallow-copy.
>
> If \`T\` has \`eltype\`, \`keytype\`, or \`valtype\` information, all elements in \`collection\` are converted to the destination type to guarantee the constraint \`collection isa T\`.
>
> If \`T\` has size or length information (e.g., \`NTuple\` and \`StaticArray\`), providing \`collection\` with unmatched size or length throws an error.
>
> Unary form \`into(T::Type)\` returns a callable \`iterable -\> into(T, iterable)\`.
>
> \## Extended help
>
> \### Example
>
> \`into\` takes care of the conversion of storage and element types:
>
> \`\`\`julia
> julia\> into(Array{Int}, BitVector(\[0, 1, 0, 0\]))
> 4-element Array{Int64,1}:
> 0
> 1
> 0
> 0
> \`\`\`
>
> \`into\` acts like a shallow copy:
>
> \`\`\`julia
> julia\> xs = Ref.(\[1, 2, 3\]);
>
> julia\> ys = into(Vector, xs)
> 3-element Array{Base.RefValue{Int64},1}:
> Base.RefValue{Int64}(1)
> Base.RefValue{Int64}(2)
> Base.RefValue{Int64}(3)
>
> julia\> ys\[1\] = Ref(100);
>
> julia\> xs
> 3-element Array{Base.RefValue{Int64},1}:
> Base.RefValue{Int64}(1)
> Base.RefValue{Int64}(2)
> Base.RefValue{Int64}(3)
>
> julia\> ys\[2\]\[\] = 200;
>
> julia\> xs
> 3-element Array{Base.RefValue{Int64},1}:
> Base.RefValue{Int64}(1)
> Base.RefValue{Int64}(200)
> Base.RefValue{Int64}(3)
> \`\`\`
>
> \`into\` \_always\_ treats input \`iterable\` as a collection:
>
> \`\`\`julia
> julia\> into(Dict, (:a =\> 1) =\> (:b =\> 2))
> Dict{Symbol,Int64} with 2 entries:
> :a =\> 1
> :b =\> 2
>
> julia\> Dict((:a =\> 1) =\> (:b =\> 2)) # but the constructor may not
> Dict{Pair{Symbol,Int64},Pair{Symbol,Int64}} with 1 entry:
> :a=\>1 =\> :b=\>2
> \`\`\`
>
> \`into(T)\` returns a function \`iterable -\> into(T, iterable)\` which is appropriate for using with \`|\>\`:
>
> \`\`\`julia
> julia\> 1:3 |\> into(NTuple{3})
> (1, 2, 3)
> \`\`\`
>
> \### Implementation
>
> The owner of the type of \`iterable\` should implement \`\_\_from\_\_\`. The owner of the output type \`T\` should implement \`\_\_into\_\_\`. If it is desirable to apply pre- and/or post-processing, the owner of the output type \`T\` may implement \`into\`.
>
>
> \# Overload API
>
> \`\`\`julia
> Base.\_\_into\_\_(T::Type, iterable) -\> collection::T
> Base.\_\_from\_\_(T::Type, iterable) -\> collection::T
> \`\`\`
>
> Overload-only API for providing the implementation for \`into(T, iterable)\`.
>
> To avoid method ambiguities, \`\_\_into\_\_\` should be implemented only by the owner of \`T\` and \`\_\_from\_\_\` should be implemented only by the owner of \`typeof(iterable)\`. The owner of \`T\` (resp. \`typeof(iterable)\`) may choose to allow \`typeof(iterable)\` (resp. \`T\`) to overload \`\_\_into\_\_\` (resp. \`\_\_from\_\_\`) by documenting specific type bounds for \`T\` (resp. \`typeof(iterable)\`).
>
> \`into(T, iterable)\` calls \`\_\_from\_\_(T, iterable)\` which in turn by default calls \`\_\_into\_\_(T, iterable)\`.
>
> \## Implementation
>
> If \`T\` is a subtype of \`AbstractArray\`,
>
> \`\`\`julia
> isequal(vec(collect′(iterable)), vec(collect(collection)))
> \`\`\`
>
> must hold where \`collect′\` is defined as
>
> \`\`\`julia
> collect′(iterable) =
> if IteratorEltype(collection) isa HasEltype
> collect(eltype(collection), iterable)
> else
> collect(iterable)
> end
> \`\`\`
>
> If \`iterable\` is a stateful iterator, \`collect\` inside \`collect′\` is "run" as if the world is rolled back to the state just before calling \`into\`.
>
> If the collections of type \`T\` do not support duplicates, \`issubset\` may be used instead of \`isequal\`. In particular, subtypes of \`AbstractDict\` and \`AbstractSet\` must satisfy the above equality with \`issubset\`.
>
> If the collections of type \`T\` do not maintain insertion order, \`issetequal\` may be used instead of \`isequal\`.
---
### Author:  [@nsajko](https://discourse.julialang.org/u/nsajko)
#### Post date: [April 30, 2024, 7:50am UTC](https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604/7 "2024-04-30T07:50:15Z")
FWIW this is the PR:
>
>
> Makes constructing \`FixedSizeArray\`s more convenient!
>
> Inspired by
> https://gi…thub.com/JuliaLang/julia/issues/36288
>
> This currently ignores \`Base.IteratorElType\`, xref https://discourse.julialang.org/t/i-dont-get-base-iteratoreltype/113604
>
> The allocations in some code paths are probably excessive/could be optimized. But I guess this is good for a start.
>
> Fixes #20