# Creating Generators

**URL:** <https://discourse.julialang.org/t/creating-generators/3962>\
**Category:** General Usage\
**Tags:** generator\
**Created:** [May 28, 2017, 12:14pm UTC](https://discourse.julialang.org/t/creating-generators/3962 "2017-05-28T12:14:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![rokkuran](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rokkuran/32/1097_2.png) [@rokkuran](https://discourse.julialang.org/u/rokkuran)\
**Post date:** [May 28, 2017, 12:14pm UTC](https://discourse.julialang.org/t/creating-generators/3962/1 "2017-05-28T12:14:17Z")

</div>

I’ve been playing around with Julia lately and have been enjoying it. I have come from python and as a learning exercise was looking at re-writing some code I had written to do least significant bit embedding.

I’m wondering if my re-write below is the best way to write a more complicated generator or if there are better or more efficient ways to implement one.

Apologies if I’ve just missed something obvious.

Original python code below:

```julia
def bit_generator(s):
    for x in s:
        a = ord(x)

        i = 0
        while i < 7:
            yield a & 1
            a = a >> 1 # bit shifting embeds character backwards
            i += 1

    # signify end with 14 zeros (double ascii null)
    for x in xrange(14):
           yield 0

```

Julia equivalent (without the nulls at the end). This is where I missed yield as I ended up creating two generators to achieve the same - I could have easily just missed something here, but this is what I came up with.

```julia
type OrdGenerator
    s::String
end

Base.start(x::OrdGenerator) = 1
Base.done(x::OrdGenerator, state) = length(x.s) == state - 1
Base.next(x::OrdGenerator, state) = Int8(x.s[state]), state + 1

type BitGenerator
    b::Int8
end

Base.start(x::BitGenerator) = 1
Base.done(x::BitGenerator, state) = state > 7

function Base.next(x::BitGenerator, state)
    a = x.b & 1
    x.b = x.b >> 1
    a, state + 1
end

s = "word? w"
println("$s : length = $(length(s))")
for (i, x) in enumerate(OrdGenerator(s))
    println("$i: $x | $(bin(x))")
    for (j, bit) in enumerate(BitGenerator(x))
        println(" $j: $bit")
    end
end

```

Any comments or advice would be appreciated.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 28, 2017, 1:12pm UTC](https://discourse.julialang.org/t/creating-generators/3962/2 "2017-05-28T13:12:53Z")

</div>

> [@rokkuran](#):
>
> I’m wondering if my re-write below is the best way to write a more complicated generator or if there are better or more efficient ways to implement one.

The best way to build a “complicated generator” is to make a type which implements the iterator interface. That’s explained very well in the manual:

[https://docs.julialang.org/en/stable/manual/interfaces/#iteration](https://docs.julialang.org/en/stable/manual/interfaces/#iteration)

---

<div class="post-metadata">

**Author:** ![rokkuran](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rokkuran/32/1097_2.png) [@rokkuran](https://discourse.julialang.org/u/rokkuran)\
**Post date:** [May 28, 2017, 3:40pm UTC](https://discourse.julialang.org/t/creating-generators/3962/3 "2017-05-28T15:40:12Z")

</div>

Thanks Chris.

I think my attempt above does use the iterator interface - happy that I’ve used the best method to create one.

After seeing your response and thinking about this a bit more, my main issue was probably more around the fact that I had to create two generators to achieve what I wanted as there was a nested while under the for.

I ended up working out that you can create a type for the state variable that has an accumulator, which works (see code below), but I’m still wondering if this approach is reasonable or what other people do?

Also, by “more complicated generator”, I really meant anything that couldn’t be easily achieved in something like a list comprehension.

```julia
type Generator
    s::String
end

type State
    i::Int64
    index_bit::Int8
    acc::Int8
end

Base.start(x::Generator) = State(1, 0, 0)
Base.done(x::Generator, state::State) = (length(x.s), 7) == (state.i, state.index_bit)

function Base.next(x::Generator, state::State)
    if state.index_bit == 0
        state = State(state.i, state.index_bit + 1, Int8(x.s[state.i]))
    elseif state.index_bit == 7
        state = State(state.i + 1, 1, Int8(x.s[state.i + 1]))
    else
        state = State(state.i, state.index_bit + 1, state.acc >> 1)
    end
    println("$(state.acc & 1) | $state")
    state.acc & 1, state
end
    
s = "here"
for (i, x) in enumerate(Generator(s)) end

```

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [May 29, 2017, 7:18am UTC](https://discourse.julialang.org/t/creating-generators/3962/4 "2017-05-29T07:18:52Z")

</div>

Have you seen `Channel`s? They are the direct Julia equivalent of Python generators.

> [@PyGen - python style generators](https://discourse.julialang.org/t/pygen-python-style-generators/3451/14):
>
> And even better (in my opinion), combining it with short-form function syntax, f(x) = Channel() do c ... push!(c, y) ... end Effectively a macro-free way of defining a python-like generator.

---

<div class="post-metadata">

**Author:** ![rokkuran](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rokkuran/32/1097_2.png) [@rokkuran](https://discourse.julialang.org/u/rokkuran)\
**Post date:** [May 29, 2017, 8:58am UTC](https://discourse.julialang.org/t/creating-generators/3962/5 "2017-05-29T08:58:08Z")

</div>

@fengyang.wang Thanks for the response.

I had looked at `Tasks` but as they were used in parallel computing, I thought they might be overkill for what I was doing.

I tried to create a generator using `Channels` (running Julia v0.5.2) this morning with no luck. I had success with the `Tasks` equivalent:

```julia
bit_generator(s::String) = Task() do
    for x in s
        a = Int8(x[1])
        i = 0
        while i < 7
            produce(a & 1)
            a = a >> 1
            i += 1
        end
    end
    
    for x in 1:14
        produce(0)
    end
end

for bit in bit_generator("text") 
    @printf("%d", bit)
end

```

My `Channels` attempt was essentially replacing as per below:

```julia
Tasks() do -> Channels() do c

```

```julia
produce(y) -> push!(c, y)

```

but I get the following error.

```julia
MethodError: Cannot `convert` an object of type ##45#46{String} to an object of type Channel{T}
This may have arisen from a call to the constructor Channel{T}(...),
since type constructors fall back to convert methods.

 in anonymous at ./<missing>:?

```

What am I missing?

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [May 29, 2017, 8:11pm UTC](https://discourse.julialang.org/t/creating-generators/3962/6 "2017-05-29T20:11:36Z")

</div>

`Task` is the right way to do this on 0.5; it is only on 0.6 that the concepts of `Channel` and `Task` have been merged.

---

<div class="post-metadata">

**Author:** ![rokkuran](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rokkuran/32/1097_2.png) [@rokkuran](https://discourse.julialang.org/u/rokkuran)\
**Post date:** [May 31, 2017, 7:35pm UTC](https://discourse.julialang.org/t/creating-generators/3962/7 "2017-05-31T19:35:55Z")

</div>

@fengyang.wang Thanks. I suspected that might be the case.

---

<div class="post-metadata">

**Author:** ![FemtoTrader](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/femtotrader/32/309_2.png) [@FemtoTrader](https://discourse.julialang.org/u/FemtoTrader)\
**Post date:** [December 16, 2017, 11:21am UTC](https://discourse.julialang.org/t/creating-generators/3962/8 "2017-12-16T11:21:06Z")

</div>

Hello,

I also looking for a way to define a generator using `Channel`

```julia
Generator1() = Channel() do c
    i = 1
    while(true)
        push!(c, i)
        i = i + 1
    end
end

function Generator2()
    c = Channel{Int}(1)
    i = 1
    while(true)
        push!(c, i)
        i = i + 1
    end
end

# println(Generator1())

for i in Generator1()
    @show i
    sleep(1)
end

```

My issue is that I don’t like `Generator1` to be a `Channel` of `Any` I’d prefer it to return a `Channel` of `Int`.

I tried to implement such an idea in `Generator2` but nothing is displayed on console.

Any idea?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [December 16, 2017, 11:32am UTC](https://discourse.julialang.org/t/creating-generators/3962/9 "2017-12-16T11:32:08Z")

</div>

```julia
Generator1() = Channel(ctype=Int) do c
           i = 1
           while(true)
               push!(c, i)
               i = i + 1
           end
       end

```

The above code is calling this constructor ` Channel(func::Function; ctype=Any, csize=0, taskref=nothing)`. The first argument comes from the do syntax and the rest of the arguments can be used normally.

---

<div class="post-metadata">

**Author:** ![Seanny123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/seanny123/32/5271_2.png) [@Seanny123](https://discourse.julialang.org/u/Seanny123)\
**Post date:** [January 14, 2019, 6:07pm UTC](https://discourse.julialang.org/t/creating-generators/3962/10 "2019-01-14T18:07:57Z")

</div>

Did you end up resolving this error?

---

<div class="post-metadata">

**Author:** ![Seanny123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/seanny123/32/5271_2.png) [@Seanny123](https://discourse.julialang.org/u/Seanny123)\
**Post date:** [January 14, 2019, 6:09pm UTC](https://discourse.julialang.org/t/creating-generators/3962/11 "2019-01-14T18:09:33Z")

</div>

In Julia 1.0, are Channels still the preferred way of implementing Python-like generators in Julia?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [January 14, 2019, 6:34pm UTC](https://discourse.julialang.org/t/creating-generators/3962/12 "2019-01-14T18:34:20Z")

</div>

Depends on your performance requirements. For the highest performance, create a type that implements the iteration protocol. If your performance demands aren’t that stringent then a channel is a fine way to go as well.

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [January 14, 2019, 8:00pm UTC](https://discourse.julialang.org/t/creating-generators/3962/13 "2019-01-14T20:00:05Z")

</div>

If you’re willing to import additional packages, [ResumableFunctions](https://github.com/BenLauwens/ResumableFunctions.jl) provides a nice way to create efficient generators.

---

<div class="post-metadata">

**Author:** ![o314](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/o314/32/252_2.png) [@o314](https://discourse.julialang.org/u/o314)\
**Post date:** [January 16, 2019, 2:43am UTC](https://discourse.julialang.org/t/creating-generators/3962/14 "2019-01-16T02:43:40Z")

</div>

Suppose you have this in python

```python
# aka riemann serie
def p_serie(p):
    assert(p>0)
    s = 0
    n = 1
    while True:
        s += 1.0/pow(n,p)
        yield s
        n += 1

```

then run

```python
itr = p_serie(2)
[next(itr) for _ in range(10)]

```

that output (reformatted)

```julia
[ 1.0,
  1.25,
  1.3611111111111112,
  1.4236111111111112,
  1.4636111111111112,
  1.4913888888888889,
  1.511797052154195,
  1.527422052154195,
  1.5397677311665408,
  1.5497677311665408
]

```

(1/2)

---

<div class="post-metadata">

**Author:** ![o314](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/o314/32/252_2.png) [@o314](https://discourse.julialang.org/u/o314)\
**Post date:** [January 16, 2019, 2:47am UTC](https://discourse.julialang.org/t/creating-generators/3962/15 "2019-01-16T02:47:04Z")

</div>

In julia, you can write

```julia
struct PSerie
    p
end

function Base.iterate(ps::PSerie, itr=(0,1,0))
    s,n,y = itr
    if y == 1; @goto y1 end

    while true
        s += 1.0/(n^ps.p)
        return s,(s,n,1)
        @label y1
        n += 1
    end
end

p_serie(n) = (x for x in PSerie(n))

```

then run

```julia
using Base.Iterators: take

[take(p_serie(2),10)...]

```

that will output:

```julia
10-element Array{Float64,1}:
 1.0
 1.25
 1.3611111111111112
 1.4236111111111112
 1.4636111111111112
 1.4913888888888889
 1.511797052154195
 1.527422052154195
 1.5397677311665408
 1.5497677311665408

```

–  
You have translated a yield using a `goto` with the help of an `iterator`.  
I like to to think that sometimes `goto` can lead to clear code.

---

<div class="post-metadata">

**Author:** ![leiteiro](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@leiteiro](https://discourse.julialang.org/u/leiteiro)\
**Post date:** [January 16, 2019, 7:10am UTC](https://discourse.julialang.org/t/creating-generators/3962/16 "2019-01-16T07:10:45Z")

</div>

Thanks for the example. What I am interested in: Is there anyone here  
who doesn’t think the Python code is easier and clearer?

ffevotte\> ‘If you’re willing to import additional packages’.

Honestly, no. This is such an elementary and frequent task that  
I think Base should provide it. Do I remember correctly that this  
worked in version 0.6?

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [January 16, 2019, 9:21am UTC](https://discourse.julialang.org/t/creating-generators/3962/17 "2019-01-16T09:21:56Z")

</div>

> [@o314](#):
>
> You have translated a yield using a `goto` with the help of an `iterator` .  
> I like to to think that sometimes `goto` can lead to clear code.

I don’t think this is a good example of that. It’s true that your solution closely resembles the original yielding python code, but for Julia developers used to Julia style iterators (and who don’t have the Python code to compare to), I think that style of iterator may be confusing. Compare your suggestion:

```julia
function Base.iterate(ps::PSerie, itr=(0,1,0))
    s,n,y = itr
    if y == 1; @goto y1 end

    while true
        s += 1.0/(n^ps.p)
        return s,(s,n,1)
        @label y1
        n += 1
    end
end

```

To the equivalent Julia style iterator:

```julia
function Base.iterate(ps::PSerie, (s,n) = (0,1))
    s += 1 / (n^ps.p)
    s, (s, n+1)
end

```

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [January 16, 2019, 11:39am UTC](https://discourse.julialang.org/t/creating-generators/3962/18 "2019-01-16T11:39:58Z")

</div>

> [@leiteiro](#):
>
> ffevotte\> ‘If you’re willing to import additional packages’.
> 
> Honestly, no. This is such an elementary and frequent task that  
> I think Base should provide it.

I’m not sure I agree anymore. I used to use generators all the time when choosing in Python, and I definitely missed this functionality when I first started working on Julia. But I can honestly say in the past year of doing basically everything in Julia, it just hasn’t been an issue.

---

<div class="post-metadata">

**Author:** ![o314](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/o314/32/252_2.png) [@o314](https://discourse.julialang.org/u/o314)\
**Post date:** [January 16, 2019, 2:26pm UTC](https://discourse.julialang.org/t/creating-generators/3962/19 "2019-01-16T14:26:38Z")

</div>

Make it  
|\> work  
|\> readable  
|\> idiomatic

🙂  
_borrowed to a great friend coder_

–

Secondly, it’s an example. so it was kept simple deliberately.

I like a lot generators too but for any functions that will have two yields or more you will got an irreducible control flow graph that you won’t be able to recode with a generator. Your solution will won’t work for two yields or more. The goto one will.

For a single case yield, use generator. They’re great

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [January 16, 2019, 2:31pm UTC](https://discourse.julialang.org/t/creating-generators/3962/20 "2019-01-16T14:31:09Z")

</div>

> [@o314](#):
>
> Make it  
> |\> work  
> |\> readable  
> |\> idiomatic

IMO, the goto solution is not more readable than the two line iterator I posted. My initial reaction is “why is there a while loop in an iterator, there must be something wrong here”. Of course, it depends on what you’re used to. Someone used to yielding iterators might find it easier to read.

> [@o314](#):
>
> I like a lot generators too but for any functions that will have two yields or more you will got an irreducible control flow graph that you won’t be able to recode with a generator. Your solution will won’t work for two yields or more. The goto one will.

Could you give an example of such a function? Perhaps there’s an alternative way to do it.

[Next page](https://discourse.julialang.org/t/creating-generators/3962.md?page=2)