# Creating Generators
**URL:** https://discourse.julialang.org/t/creating-generators/3962
**Category:** General Usage
**Tags:** generator
**Created:** [May 28, 2017, 12:14pm UTC](https://discourse.julialang.org/t/creating-generators/3962 "2017-05-28T12:14:17Z")
**Posts on this page:** 13
**Page:** 2
### Author:  [@o314](https://discourse.julialang.org/u/o314)
#### Post date: [January 16, 2019, 2:35pm UTC](https://discourse.julialang.org/t/creating-generators/3962/21 "2019-01-16T14:35:50Z")
> [@leiteiro](#):
>
> Thanks for the example. What I am interested in: Is there anyone here
> who doesn’t think the Python code is easier and clearer?
I think yield is a very good construct, good reasons they have bloomed here and there in so many languages.
IMHO they weren’t tackled enough seriously in Julia v1, that has so many other great features too.
After all, they have landed in C# 2 not 1.
---
### Author:  [@Orbots](https://discourse.julialang.org/u/Orbots)
#### Post date: [January 16, 2019, 7:53pm UTC](https://discourse.julialang.org/t/creating-generators/3962/22 "2019-01-16T19:53:08Z")
> [@bennedich](#):
>
> My initial reaction is “why is there a while loop in an iterator, there must be something wrong here”
I’d add one more to the list.
|\> extensible
In this case they’ve made the state machine explicit. Iterator protocol has an implicit state machine. This goto/while/return construct can be extended to handle more complex state transitions.
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [January 16, 2019, 8:06pm UTC](https://discourse.julialang.org/t/creating-generators/3962/23 "2019-01-16T20:06:26Z")
> [@leiteiro](#):
>
> Is there anyone here
> who doesn’t think the Python code is easier and clearer?
I guess this is mostly a matter of getting used to some pattern, which then feels natural (to a point where one even forgets that one had to learn the concept in the first place).
I remember the first time I saw a Python generator using the `yield` statement: I did not know what it was doing, and had trouble to understand how it worked and why/when one would want to use it. After some time however, it felt natural. When I started using Julia, which had the same kind of concepts at the time, I felt I was not in uncharted territories in this respect.
Then newer versions of Julia started using `Channels` to define coroutines, and this is not as efficient for generators. It did put me off-balance for some time, but now I came to get used to writing `iterate` functions.
In practice, I find that composing existing iterators (coming from [`Base.Iterators`](https://docs.julialang.org/en/v1/base/iterators/) or built with the special comprehension-like syntax for generators) goes a long way. Composing [`Transducers`](https://github.com/tkf/Transducers.jl) is now another option. I have come to think that writing custom `iterate` methods for more specific things is not so hard (although it requires being in a different state of mind than writing coroutines).
---
### Author:  [@bennedich](https://discourse.julialang.org/u/bennedich)
#### Post date: [January 16, 2019, 8:25pm UTC](https://discourse.julialang.org/t/creating-generators/3962/24 "2019-01-16T20:25:49Z")
> [@Orbots](#):
>
> I’d add one more to the list… extensible… This goto/while/return construct can be extended to handle more complex state transitions.
Do you have an example of a complex generator implemented with a goto/while/return construct? Not trying to argue, I’m genuinely interested if there’s a clean, readable and extensible goto-based solution to this that I’ve missed. Having a concrete example to discuss around would be helpful.
---
### Author:  [@Orbots](https://discourse.julialang.org/u/Orbots)
#### Post date: [January 16, 2019, 9:07pm UTC](https://discourse.julialang.org/t/creating-generators/3962/25 "2019-01-16T21:07:33Z")
I personally wouldn’t use this style so I can’t think of an example off the top of my head.
The goto let’s you jump back to the code ( state ) you returned from. The while is one of those state transitions that goes back to itself ( can have nested states here ). Here’s an example of an iterator with some explicit state machine stuff going on: [Shawn Hargreaves Blog Index](https://blogs.msdn.microsoft.com/shawnhar/2010/10/01/iterator-state-machines/)
You could make iterators as arbitrarily complex as you like by making your state data as complex as you like. Or you can push some of that complexity into logic ( e.g. goto/while/return ) instead.
Usually I would run into problems where I need to explicitly model a state machine in code when there is UI involved. This can end up being very modal ( nested state machines ).
I guess if you wanted to implement some kind of FRP style with iterators you could end up needing some goto’s: [Functional reactive programming - Wikipedia](https://en.wikipedia.org/wiki/Functional_reactive_programming)
My choice in that case would be to use Channels. CSP [Communicating sequential processes - Wikipedia](https://en.wikipedia.org/wiki/Communicating_sequential_processes) is great, and I’d love to see a select/choose/alts ( everyone has their own name! ) function for Julia’s channels.
[https://github.com/JuliaLang/julia/issues/13763](https://github.com/JuliaLang/julia/issues/13763)
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [January 16, 2019, 11:04pm UTC](https://discourse.julialang.org/t/creating-generators/3962/26 "2019-01-16T23:04:06Z")
> [@bennedich](#):
>
> Do you have an example of a complex generator implemented with a goto/while/return construct?
I guess an (entirely made-up) example of a generator with multiple `yield` statements could be something like this:
## Version using `yield`
```julia-auto
using ResumableFunctions
@resumable function myIter()
i = 1
while true
if iseven(i)
@yield i
else
for j in 1:i
@yield -j
end
end
i += 1
end
end
```
```julia-auto
julia> Iterators.take(myIter(), 10) |> collect
10-element Array{Any,1}:
-1
2
-1
-2
-3
4
-1
-2
-3
-4
```
## Version using `goto` / `while` / `return`
```julia-auto
struct MyIterGoto end
function Base.iterate(::MyIterGoto, (i, j, l) = (0,0,0))
if l == 1; @goto l1
elseif l == 2; @goto l2
end
i = 1
while true
if iseven(i)
return i, (i,0,1)
@label l1
else
j = 1
while j <= i
return -j, (i,j,2)
@label l2
j += 1
end
end
i += 1
end
end
Base.IteratorSize(::Type{MyIterGoto}) = Base.SizeUnknown()
```
## “unrolled” version
```julia-auto
struct MyIterUnrolled end
function Base.iterate(::MyIterUnrolled, (i,j) = (1,1))
if iseven(i)
return i, (i+1, 1)
elseif j == i
return -j, (i+1, 1)
else
return -j, (i, j+1)
end
end
Base.IteratorSize(::Type{MyIterUnrolled}) = Base.SizeUnknown()
```
Although this version is very concise, I guess it is also arguably much harder to read and understand.
* * *
## [EDIT] Transducer-based version
```julia-auto
using Transducers
t = MapCat(i -> iseven(i) ? (i,) : -(1:i))
collect(t |> Take(10), Iterators.countfrom(1))
```
This version based on [Transducers.jl](https://tkf.github.io/Transducers.jl/dev/) is also very compact, while (IMHO) remaining easy to understand.
---
### Author:  [@Orbots](https://discourse.julialang.org/u/Orbots)
#### Post date: [January 16, 2019, 11:25pm UTC](https://discourse.julialang.org/t/creating-generators/3962/27 "2019-01-16T23:25:58Z")
Key thing to notice is that the last version duplicates the `-j` logic. A function call here would still result in two call sites vs one. The state transition logic is also complected with the computation for the value of `j`.
The state machine logic is more explicit in the first two.
So, I’d argue that it’s a more extensible pattern because it isolates the new behaviour in one place. Well there are two more places for the transition logic, but that is structured nicely.
---
### Author:  [@bennedich](https://discourse.julialang.org/u/bennedich)
#### Post date: [January 17, 2019, 12:40am UTC](https://discourse.julialang.org/t/creating-generators/3962/28 "2019-01-17T00:40:59Z")
> [@Orbots](#):
>
> Key thing to notice is that the last version duplicates the `-j` logic.
That seems easily fixed:
```julia
function Base.iterate(::MyIterUnrolled, (i,j) = (1,1))
if iseven(i)
value = i
new_state = (i=i+1, j=1)
else
value = -j
new_state = j < i ? (i=i, j=j+1) : (i=i+1, j=1)
end
return value, new_state
end
```
---
### Author:  [@Orbots](https://discourse.julialang.org/u/Orbots)
#### Post date: [January 17, 2019, 1:14am UTC](https://discourse.julialang.org/t/creating-generators/3962/29 "2019-01-17T01:14:04Z")
> [@bennedich](#):
>
> That seems easily fixed
Right. So the thing I see now that is different is that the continuation versions don’t need to recheck the state condition `iseven(i)` to deterimine which state it is in. They are explicitly tracking which state it’s in and directly jump to it via the `gotos` at the top. You could easily fix that as well without the `gotos` by tracking `l` ( which state machine node we are in )
```
function Base.iterate(::MyIterUnrolled, (i,j,l) = (1,1,0))
if l == 0
...
elseif l == 1
....
else # l == 2
end
```
Then you end up with multiple call sites for the `-j` mapping again. The continuation style seems less error prone, but that’s likely a personal preference.
[EDIT]
personal preference would be to use channels, which allow you to write very linear looking logic.
```
ijchan(n) = Channel(c-> map( i->iseven(i) ? put!(c,i) : map(j->put!(c,-j),1:i), 1:n))
```
---
### Author:  [@Olof\_Salberger](https://discourse.julialang.org/u/Olof_Salberger)
#### Post date: [February 4, 2019, 10:01am UTC](https://discourse.julialang.org/t/creating-generators/3962/30 "2019-02-04T10:01:08Z")
Especially if you use do-blocks and unroll it over several lines instead of lambdas & ternary. Using map for code with side effects is frowned upon, I would definitely use a do-block and for-loops here.
```julia
jchan(n) = Channel(ctype=Int) do c
for i = 1:n
if iseven(i)
put!(c,i)
else
for j = 1:i
put!(c,-j)
end
end
end
end
```
Which indeed, is exactly like the code with yield in Python, except it’s more powerful since you have the full power of coroutines if you want to do something more general. It’s slow by Julia standards, but very fast by Python standards (coroutine yield in Julia is generally faster than a plain function call in Python, let alone a generator call) so that’s not an issue when comparing with Python.
---
### Author:  [@zot](https://discourse.julialang.org/u/zot)
#### Post date: [December 9, 2021, 4:11pm UTC](https://discourse.julialang.org/t/creating-generators/3962/31 "2021-12-09T16:11:38Z")
I separated a very small and lightweight generator package out of one of my projects – this just uses tasks, not channels:
[https://github.com/zot/Generators.jl](https://github.com/zot/Generators.jl)
Hopefully the package will get registered soon…
---
### Author:  [@Kirby\_Zhang](https://discourse.julialang.org/u/Kirby_Zhang)
#### Post date: [June 6, 2022, 9:39pm UTC](https://discourse.julialang.org/t/creating-generators/3962/32 "2022-06-06T21:39:16Z")
ResumableFunctions.jl is really good, some day it should enter the Base package. It’s also _fast_, much faster than the methods using tasks and channels. I’ve tried Python generators implemented in Numba, they’re very slow while rest of Numba code is good performance. So this is not easy. ResumableFunctions is close to direct access performance, according to their tests.
---
### Author:  [@xor0110](https://discourse.julialang.org/u/xor0110)
#### Post date: [July 6, 2022, 8:34pm UTC](https://discourse.julialang.org/t/creating-generators/3962/33 "2022-07-06T20:34:58Z")
This post isn’t very recent, but it showed up in my suggestions here, and I’m very interested in the topic. I’m a big fan of do-syntax and the ability of creating a generator/iterator using it. Unfortunately there are some gotchas with that, `Channel` will give you asynchronous behavior, but you don’t necessarily want that. The interface is great, though. That’s one big reason I proposed this PR. Still not merged, but check it out: [https://github.com/JuliaLang/julia/pull/44873](https://github.com/JuliaLang/julia/pull/44873)
[Previous page](https://discourse.julialang.org/t/creating-generators/3962.md?page=1)