Iterators.Stateful is not an iterator?

New to Julia
1

I’m trying to understand iterators better because I’m trying to decide whether I should, for my specific application, define iterate for my type or define a constructor of Iterators.Stateful for my type. And I seem to have reached the conclusion that Iterators.Stateful aren’t iterators?

The interfaces page [1] says that to implement the interator interface, two methods are required:

  • iterate(iter) Returns either a tuple of the first item and initial state or nothing if empty
  • iterate(iter, state) Returns either a tuple of the next item and next state or nothing if no items remain

So a vector is an iterator:

x = ["a", "b", "c"];
iterate(x) # returns ("a", 2) - state 2 indexes the next element
iterate(x, 2) # returns ("b", 3)
iterate(x, 3) # returns ("c", 4)
iterate(x, 4) # returns nothing

What happens if we make a stateful iterator out of x?

it = Iterators.Stateful(x);
iterate(it) # returns ("a", nothing) - no state?
iterate(it) # returns ("b", nothing) - no state?
iterate(it) # returns ("c", nothing) - no state?
iterate(it) # returns nothing - this one is correct

As a separate point, I’ve found this git issue which seems to show an enormous bug: https://github.com/JuliaLang/julia/issues/35530

So my question is: Am I misunderstanding something? And if not, should I conclude that I shouldn’t use Iterators.Stateful as it looks kind of broken / abandoned?

2

No, this is definitely expected behavior. Stateful stores the iteration state inside the Stateful object, so there is no reason to pass any additional state around here.

1 Like
3

I understand there’s no reason to, but it still doesn’t agree with interface spec…

4

Yes, this is a bug, as pointed out in the issue, since collect fails to take into account that an iterator with defined length could also be stateful

1 Like
5

It perfectly agrees with that for me:

julia> it = Iterators.Stateful([1, 2, 3]);

julia> iterate(it)
(1, nothing)

julia> iterate(it, ans[2])
(2, nothing)

julia> iterate(it, ans[2])
(3, nothing)

julia> iterate(it, ans[2]) === nothing
true
2 Likes
6

Ok fine different readings. I read “Returns either a tuple of the next item and next state” as meaning that the “next state” actually controls what is returned by applying iterate(x, state), you read it as meaning that’s not necessarily the case.

1 Like
7

Yes, iterate doesn’t necessarily have to be pure, so you always need state as well as the object (and perhaps even additional global state, as in eachline()) to predict what the next item will be and the object may also be mutated during iterate. Perhaps the docs could be a bit clearer here, feel free to make a PR, if you think this could be explained better.

4 Likes
8

Exactly, that’s another point. When I first read “Returns either a tuple of the next item and next state” I took it as implying “… and therefore the object doesn’t have to be mutated”. Now it doesn’t surprise me that this aspect is false for the case of stateful iterators - their name indicates as much. But now that you spelled it out, I realized: since iterate doesn’t contain an exclamation mark ! in its name, I implicitly assumed it wouldn’t mutate (again, expect for stateful iterators).

Anyway, thanks for the clarification.

9

Thank you @freeman and @simeonschaub for the explanations. I was also confused by the fact that iterate(x, state) doesn’t have ! as @freeman said. And I admit that I wasn’t careful enough to read the following sentences in the page (emphasis mine).

Instead of mutating objects as they are iterated over, Julia iterators may keep track of the iteration state externally from the object.

To enhance the doc, how about adding:

state may or may not be used depending on iter.

to the “Brief description” of iterate(x, state)?

And add the following example (taken from @simeonschaub 's)

julia> it = Iterators.Stateful([1, 2, 3]);

julia> iterate(it, nothing)
(1, nothing)

julia> iterate(it, nothing)
(2, nothing)

julia> iterate(it, nothing)
(3, nothing)

If the general direction is OK, I’ll prepare a PR for the doc.

1 Like