From generator (with Channel) to iterator

General Usage
1

Hello,

PyGen - python style generators - #17 by nsmith and Creating Generators introduce how to mimics Python generator with Julia Channel.

With Python, a generator can be used like an iterator with next function (or __next__ method)

In [1]: def gen():
   ...:     for i in  range(10):
   ...:         yield i
   ...:         

In [2]: for i in gen():
   ...:     print(i)
   ...:     
0
1
2
3
4
5
6
7
8
9

In [3]: g = gen()

In [4]: next(g)  # or g.__next__()
Out[4]: 0

In [5]: next(g)
Out[5]: 1

In [6]: next(g)
Out[6]: 2

In [7]: next(g)
Out[7]: 3

In [8]: next(g)
Out[8]: 4

In [9]: next(g)
Out[9]: 5

In [10]: next(g)
Out[10]: 6

In [11]: next(g)
Out[11]: 7

In [12]: next(g)
Out[12]: 8

In [13]: next(g)
Out[13]: 9

In [14]: next(g)
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-14-5f315c5de15b> in <module>()
----> 1 next(g)

StopIteration:

I wonder if a Julia generator implemented using Channel can be used like an iterator (ie have a next function implementation)

julia> gen() = Channel(ctype=Int) do c
         for i in 0:10-1
           push!(c, i)
         end
       end
gen (generic function with 1 method)

julia> g = gen()

julia> for i in g
         println(i)
       end
0
1
2
3
4
5
6
7
8
9

Unfortunatelly it doesn’t work as expected:

julia> g = gen()

julia> state = start(g)
Base.ChannelIterState{Int64}(false, 0)

julia> i, state = next(g, state)
(0, Base.ChannelIterState{Int64}(false, 0))

julia> i, state = next(g, state)
(0, Base.ChannelIterState{Int64}(false, 0))

julia> i, state = next(g, state)
(0, Base.ChannelIterState{Int64}(false, 0))

Any idea why it’s behaving this way?

Kind regards

2

If you look at the code (https://github.com/JuliaLang/julia/blob/master/base/channels.jl), you will see that done is where the next value in the Channel is extracted. So before every next call, you have to call done(g, state). Pretty sure for loops call done before every call to next.

3

Fixing this is precisely what the iteration protocol change mentioned here is about.

4

You are right

julia> g = gen()
Channel{Int64}(sz_max:0,sz_curr:1)

julia> state = start(g)
Base.ChannelIterState{Int64}(false, 0)

julia> done(g, state)
false

julia> i, state = next(g, state)
(0, Base.ChannelIterState{Int64}(false, 0))

julia> done(g, state)
false

julia> i, state = next(g, state)
(1, Base.ChannelIterState{Int64}(false, 1))

julia> done(g, state)
false

julia> i, state = next(g, state)
(2, Base.ChannelIterState{Int64}(false, 2))

julia> done(g, state)
false

julia> i, state = next(g, state)
(3, Base.ChannelIterState{Int64}(false, 3))

julia> done(g, state)
false

julia> i, state = next(g, state)
(4, Base.ChannelIterState{Int64}(false, 4))

julia> i, state = next(g, state)
(4, Base.ChannelIterState{Int64}(false, 4))

julia> i, state = next(g, state)
(4, Base.ChannelIterState{Int64}(false, 4))

julia> i, state = next(g, state)
(4, Base.ChannelIterState{Int64}(false, 4))

julia> done(g, state)
false

julia> i, state = next(g, state)
(5, Base.ChannelIterState{Int64}(false, 5))

so if I understand correctly @StefanKarpinski with Julia 0.7 (or 1.0?), calling next (without done) will be enough, isn’t it?

5

Not yet, but once @keno merges his changes, instead of start, next and done there will only be iterate:

  • iterate(itr) will replace start(itr)
  • iterate(itr, state) will replace done(itr, state) and next(itr, state):
    • if it returns nothing then the iterator is done;
    • if it returns a (value, state) pair then it’s not.

This allows iterators like channels to try getting a new value and decide after trying whether they’re done or not.

10 Likes
6

I wonder if this discussion have ever been linked with one with Do While Loop such as Redirecting to Google Groups because this it typically a case where you need to run a conditional instruction after one loop.

1 Like
7

I noticed that Stateful has recently been added to Base.Iterators (not available in v6.2.2 :neutral_face:)
It implements an iterator that knows about its own state.
Question: what is the reason regular iterators externalize their state? Or, said differently, what would we lose if all iterators were Stateful and hide their state?

1 Like