Generic function for getting next item from stateful iterator

General Usage
1

Say I have a type that satisfies the iterator interface. It’s a stateful iterator that behaves similarly to StatefulIterator, so every time it generates an item the state changes internally.

What’s the right generic function to use to get the next item? StatefulIterator uses popfirst!, which seems reasonable, but then if I try to wrap my iterator in a generator the generator doesn’t implement popfirst!:

MWE:

mutable struct MyIter
    next::Int
    max::Int
    MyIter() = new(0, rand(1:10))
end

Base.IteratorSize(::Type{MyIter}) = Base.SizeUnknown()
Base.eltype(::MyIter) = Int
# internal function that gets the next item and mutates internal state
function getnext(m::MyIter)
    if m.next > m.max
        nothing
    else
        val = m.next
        m.next += 1
        val
    end
end

function Base.iterate(m::MyIter, state=nothing)
    next = getnext(m)
    next === nothing ? nothing : (next, nothing)
end

# could this instead be implemented automatically by an iterator fallback?
Base.popfirst!(m::MyIter) = getnext(m)

m = MyIter()
# this works
popfirst!(m)

# this gives MethodError because `Base.Generator` doesn't have a `popfirst!` method
wrapper = (2x for x in m)
popfirst!(wrapper)

It feels like maybe I’m swimming upstream a little bit treating this stateful thing as an iterator, but the fact that StatefulIterator exists makes me think this isn’t a totally crazy thing to do.

Would it make sense for there to be a popfirst! fallback in terms of iterate?

2

For a stateful iterator, iterate and popfirst! are supposed to be identical, except that the latter should error when there is no element, rather than returning nothing. You’ll also want to implement isdone for your stateful iterator to make sure zip works correctly. Stateful iterators aren’t super well supported in Base at the moment, but they do at least work better now than they used to. I’m not really sure that popfirst! should work on the generator. If so, the definition would of course have to be:

popfirst!(a::Generator) = a.f(popfirst!(a.iter))

i.e. be separate from the iteration interface. That would be a fine definition, I think, but somewhat orthogonal to iteration. However, if you do want to use the iteration interface iterate(a)[1] works just fine even with the wrapper. popfirst! is just a bit cleaner and more explicit.

1 Like
3

Also different in that when there is an element iterate returns both the element and state (presumably nothing for a stateful iterator), and popfirst! would just return the element.

I just realized that take! seems like a pretty natural choice, as well. It looks like StatefulIterator switched from take! to popfirst!, but it’s not clear to me how the semantics of those two functions are supposed to be different.