# Is it always safe to “re-use” an iterator/generator?
**URL:** https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697
**Category:** General Usage
**Tags:** generator, iterators
**Created:** [March 7, 2023, 9:39pm UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697 "2023-03-07T21:39:29Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)
#### Post date: [March 7, 2023, 9:39pm UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/1 "2023-03-07T21:39:29Z")
The way iterators work in Python, there is the following (potentially surprising) behavior:
```julia
>>> l = [1, 2, 3]
>>> g = (x for x in l)
>>> list(g)
[1, 2, 3]
>>> list(g)
[]
```
That is, iterators are “used up” during the iteration and will not return items again which have already been iterated over.
Unfortunately, I haven’t been able to find much detailed documentation about the behavior of iterators in Julia. The main parts are in
- [Interfaces · The Julia Language](https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-iteration)
- [Collections and Data Structures · The Julia Language](https://docs.julialang.org/en/v1/base/collections/#lib-collections-iteration)
and focus more on how to define new iterators, not on what the guarantees on the behavior should be. Since (contrary to Python) the iteration state is kept track of as a separate variable outside the iterator object itself in Julia, presumably it should always be possible to iterate an iterator repeatedly (since it should probably return the same value given the same state parameter):
```julia
julia> l = [1, 2, 3]
3-element Vector{Int64}:
1
2
3
julia> g = (x for x in l)
Base.Generator{Vector{Int64}, typeof(identity)}(identity, [1, 2, 3])
julia> collect(g)
3-element Vector{Int64}:
1
2
3
julia> collect(g)
3-element Vector{Int64}:
1
2
3
```
But is this really _always_ the case?
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [March 7, 2023, 9:44pm UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/2 "2023-03-07T21:44:05Z")
> [@lmtzx9h4qqnt](#):
>
> But is this really _always_ the case?
I think there are a few exceptions for cases that involve iterating over streaming data. e.g. the `eachline` iterator is [documented](https://docs.julialang.org/en/v1/base/io-network/#Base.eachline) as closing the file at the end of iteration, so it is `isempty` if you try to iterate a second time.
---
### Author:  [@simsurace](https://discourse.julialang.org/u/simsurace)
#### Post date: [March 7, 2023, 9:45pm UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/3 "2023-03-07T21:45:51Z")
Also iterating over a `Channel` will take from it, but that is probably expected.
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [March 7, 2023, 10:10pm UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/4 "2023-03-07T22:10:01Z")
Also see [Iterators.Stateful](https://docs.julialang.org/en/v1/base/iterators/#Base.Iterators.Stateful), which turns any iterator into a “stateful” iterator (i.e. an iterator which behaves more like a `Channel` or a Python iterator).
For example:
```julia
julia> l = [1, 2, 3]
3-element Vector{Int64}:
1
2
3
```
a generator is stateless (as is the vector `l` itself)
```julia
julia> g = (x for x in l)
Base.Generator{Vector{Int64}, typeof(identity)}(identity, [1, 2, 3])
julia> collect(g)
3-element Vector{Int64}:
1
2
3
julia> collect(g)
3-element Vector{Int64}:
1
2
3
```
but we can build a stateful iterator out of it:
```julia
julia> h = Iterators.Stateful(g)
Base.Iterators.Stateful{Base.Generator{Vector{Int64}, typeof(identity)}, Union{Nothing, Tuple{Int64, Int64}}}(Base.Generator{Vector{Int64}, typeof(identity)}(identity, [1, 2, 3]), (1, 2), 0)
julia> collect(h)
3-element Vector{Int64}:
1
2
3
julia> collect(h)
Int64[]
```
---
### Author:  [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)
#### Post date: [March 7, 2023, 10:12pm UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/5 "2023-03-07T22:12:28Z")
Indeed, “proof by counter-example” 😃
```julia
julia> s = "a
b
c
d"
"a\nb\nc\nd"
julia> el = eachline(IOBuffer(s))
Base.EachLine{IOBuffer}(IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=7, maxsize=Inf, ptr=1, mark=-1), Base.var"#408#411"(), false)
julia> collect(el)
4-element Vector{String}:
"a"
"b"
"c"
"d"
julia> collect(el)
String[]
```
Thanks!
---
### Author:  [@mbaz](https://discourse.julialang.org/u/mbaz)
#### Post date: [March 7, 2023, 10:21pm UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/6 "2023-03-07T22:21:35Z")
> [@lmtzx9h4qqnt](#):
>
> focus more on how to define new iterators, not on what the guarantees on the behavior should be
There are no guarantees, because iterators can be inifinite and/or stateful, and the only way to find out is read the docs/source or experiment.
The good news is that (as far as I’m aware) iterators of a given type all behave the same. For example, `Base.Generator` always behaves as in your example.
> [@lmtzx9h4qqnt](#):
>
> the iteration state is kept track of as a separate variable outside the iterator object itself
Not necessarily; for a stateful iterator, it may make more sense to keep the state internally. See an example [here](https://github.com/mbaz/CommunicationsSequences.jl/blob/0d9434baf75a63ea7eb14387413f879fd1c0ab0d/src/lfsr.jl#L78).
---
### Author:  [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)
#### Post date: [March 7, 2023, 10:32pm UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/8 "2023-03-07T22:32:23Z")
Well, it’s definitely possible to document whatever guarantees there should be as part of the interface specification. The way the documentation is structured, it just didn’t seem clear to me one way or the other.
Infinite iterators don’t cause problems here in principle – since their values obviously can’t be stored in any kind of memory, they’re always computed “on the fly” anyway, which can be kept track of using the state variable. But when it comes to what I’ve learned now are “stateful iterators”, it’s a matter of definition whether that behavior violates the iterator interface or not. Given a constant value of `state`, I’d certainly find it reasonable to expect `iterate(iter, state)` to always return the same value – but as you and others show, this doesn’t seem to be a requirement.
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [March 8, 2023, 10:47am UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/9 "2023-03-08T10:47:51Z")
> [@lmtzx9h4qqnt](#):
>
> Well, it’s definitely possible to document whatever guarantees there should be as part of the interface specification.
There are indeed proposals to formalize the guarantees that iterators provide via traits. See for example this issue and the links therein:
>
>
> Currently, Julia generally assumes iterators are stateless. This is clearly wron…g, and so leads to issues:
>
> \* https://github.com/JuliaLang/julia/issues/43235
> \* https://github.com/JuliaLang/julia/issues/42107
> \* https://github.com/JuliaLang/julia/issues/42168
> \* https://github.com/JuliaLang/julia/issues/27412
>
> I have run into some of these issues myself, and they cause bugs that are quite annoying to track down.
> Also, a few days ago, @kescobo reported this:
> \`\`\`
> julia\> lines = (i for i in eachline(IOBuffer("a\\nb")));
>
> julia\> iterate(lines)
> ("a", nothing)
>
> julia\> isempty(lines)
> false
>
> julia\> iterate(lines)\[1\]
> ERROR: MethodError: no method matching getindex(::Nothing, ::Int64)
> \`\`\`
>
> Previously, these issues has been resolved by the band-aid approach: Wait until the erroneous assumption that iterators are stateless causes problems, then special-case those problems.
> I think @kescobo 's issue is a good example why this is not viable: The number of possible stateful iterators are infinite. We need a more principled approach.
>
> I propose we make a trait that tells if an iterator has state or not. It defaults to \`IsStateless\`. We implement \`IsStateful\` for a few basic stateful iterators, and then iterators built on top of other stateful iterators are also stateful.
> We can then make functions like \`isempty\` error for stateful iterators, unless \`isdone\` has been defined.
>
> @oxinabox proposed something similar in https://github.com/JuliaLang/julia/issues/27412.
---
### Author:  [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)
#### Post date: [March 15, 2023, 2:21am UTC](https://discourse.julialang.org/t/is-it-always-safe-to-re-use-an-iterator-generator/95697/10 "2023-03-15T02:21:05Z")
A relevant previous discussion:
> [@Iterators.Stateful is not an iterator?](https://discourse.julialang.org/t/iterators-stateful-is-not-an-iterator/53080):
>
> I’m trying to understand iterators better because I’m trying to decide whether I should, for my specific application, define iterate for my type or define a constructor of Iterators.Stateful for my type. And I seem to have reached the conclusion that Iterators.Stateful aren’t iterators? The interfaces page [[1]](https://docs.julialang.org/en/v1/manual/interfaces/) says that to implement the interator interface, two methods are required: iterate(iter) Returns either a tuple of the first item and initial state or [nothing](https://docs.julialang.org/en/v1/base/constants/#Core.nothing) if empty iterate(iter, …