Base.take does not respect that the argument has SizeUnknown()? (0.5.0)

General Usage
1

At least Base.take and Base.drop seem to ignore that their argument iterator has SizeUnknown(), which in turn makes collect(take(...)) crash with “MethodError: no method matching length” when there in fact is no length method for the argument. I found related issues from 2015, when I think Base.iteratorsize was not there yet, but is this still an issue, or am I doing something wrong here?

I’m playing with the precompiled Julia 0.5.0 (64-bit generic Linux) binary to see how well the new generator machinery now works. The point of this test case is that it has random length.

module GenerationExploration

export Pool
import Base: start, done, next, iteratorsize

immutable Pool ; data end

Base.start(pool::Pool) = nothing
Base.done(pool::Pool, state) = rand(1:10) == 1
Base.next(pool::Pool, state) = rand(pool.data), state
Base.iteratorsize(::Pool) = Base.SizeUnknown()

# julia> using GenerationExploration
# 
# julia> Base.iteratorsize(Pool(['3', '1']))
# Base.SizeUnknown()
# 
# julia> Base.iteratorsize(take(Pool(['3', '1']), 2))
# Base.HasLength()
#
# Why does Base.take think it HasLength() when a Pool clearly tells it
# that it has not?

end # module
2

Pretty sure it should be:

Base.iteratorsize(::Type{Pool}) = Base.SizeUnknown()

I did the same mistake before in a package, which got fixed by a PR :slight_smile: (fix methods of iterator interface by bicycle1885 · Pull Request #14 · JuliaLang/Tokenize.jl · GitHub).

3

Aha! Yes, that does it! Much thanks indeed.

I added also eltype and now things are nice again. (But I do have a feeling that this is not the last time I’m going to be blind to this same mistake. Awkward.)

module GenerationExploration

export Pool
import Base: start, done, next
import Base: iteratorsize, iteratoreltype, eltype

immutable Pool{T} ; data::Array{T} end

Base.start(pool::Pool) = nothing
Base.done(pool::Pool, state) = rand(1:10) == 1
Base.next(pool::Pool, state) = rand(pool.data), state
Base.iteratorsize{T}(::Type{Pool{T}}) = Base.SizeUnknown()
Base.iteratoreltype{T}(::Type{Pool{T}}) = Base.HasEltype()
Base.eltype{T}(::Type{Pool{T}}) = T

# now it works with Base.take and Base.drop
end # module
4

As a comment, there is no need for the {T} stuff in

Base.iteratorsize{T}(::Type{Pool{T}}) = Base.SizeUnknown()

since you are not using T either in the function body or for dispatch. Type{Pool} will work just as well.

5

Thanks, but the {T} in iteratorsize seems necessary. Without it, I observe that iteratorsize reports that a Pool now HasLength(), which is the default.

Am I using the type parameter for dispatch, or am I not?

I can remove {T} from my iteratoreltype without breaking my current tests. Not sure what the difference is.

6

Yes, sorry, my mistake! Types are not covariant so I was indeed wrong.

7

Perhaps

Base.iteratorsize{T<:Pool}(::Type{T}) = Base.SizeUnknown()
1 Like