# PyGen - python style generators

**URL:** <https://discourse.julialang.org/t/pygen-python-style-generators/3451>\
**Category:** Community\
**Tags:** package, announcement\
**Created:** [May 1, 2017, 4:09am UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451 "2017-05-01T04:09:29Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 1, 2017, 4:09am UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/1 "2017-05-01T04:09:29Z")

</div>

I am addicted to those `yield` statements in python. If you’re in the same boat checkout [PyGen](https://github.com/nsmith5/PyGen). I know you can achieve the same thing with `Channels` but its always felt a little clunky to me. Anyways, here is how it works:

```julia
julia> using PyGen

julia> @pygen """ 
function fibonacci()
    n, m = 0, 1
    while true
        yield m
        n, m = m, n + m
    end
end
"""
fibonacci (generic function with 1 method)

julia> for i in fibonacci()
           println(i)
           sleep(1)
       end
1
1
3
5
8
.
.
.

```

Enjoy 🙂

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [May 1, 2017, 2:55pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/2 "2017-05-01T14:55:59Z")

</div>

Cool! Have you considered writing a regular macro, instead of one operating on strings?

```julia
@pygen function fibonacci()
    n, m = 0, 1
    while true
        yield(m)
        n, m = m, n + m
    end
end

```

will be more robust, and won’t kill auto-indent/syntax highlighting. You might find [MacroTools’ expression walking functions](https://github.com/MikeInnes/MacroTools.jl#expression-walking) useful.

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 1, 2017, 3:05pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/3 "2017-05-01T15:05:59Z")

</div>

Oh good call, I did it this way because the REPL would barf as soon as I wrote any `yield x` statements, but the `yield(x)` fixes that. [MacroTools](https://github.com/MikeInnes/MacroTools.jl) looks heavenly, thanks for the advice 🙂

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [May 1, 2017, 4:10pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/4 "2017-05-01T16:10:24Z")

</div>

If you don’t want to use `yield(m)` but have more of a statement feel, you can use `@yield m`. You don’t ever have to define a `@yield` macro if you make sure to handle and eliminate those `@yield` `Expr`s in the `pygen` macro.

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 1, 2017, 6:14pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/5 "2017-05-01T18:14:10Z")

</div>

Thanks for the tips everyone! `@pygen` is even better than before :).

```julia
julia> using PyGen

julia> @pygen function pascal(n)
           i = 0
           while i <= n
               yield(i)
               i += 1
           end
       end
pascal (generic function with 1 method)

julia> n = 20;

julia> sum(pascal(n)) == n * (n + 1) / 2 # Booyah computers work
true 

```

I think I’m beginning to see the light on the whole macro business… Feels good to just add the features you want to a language.

---

<div class="post-metadata">

**Author:** ![dpsanders](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dpsanders/32/3573_2.png) [@dpsanders](https://discourse.julialang.org/u/dpsanders)\
**Post date:** [May 1, 2017, 8:23pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/6 "2017-05-01T20:23:38Z")

</div>

The name “PyGen” suggests that it is something to do with Python, whereas it isn’t really. Maybe something like “yieldgen”?

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 1, 2017, 9:11pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/7 "2017-05-01T21:11:47Z")

</div>

I suppose as a package name it sort of clashes with the current trend of being a wrapper of a Python library or Python itself, but I don’t see why implementing a piece of syntax from Python is very different. Unfortunately, the vocabulary on the Julia side makes hard to talk about it without reference to Python.

`@yield`, `@generator` etc already have specific meaning in Julia… perhaps `@pyyield` because it supplies the Python type `yield` statement?

I do agree it has a troubled sort of name (both the macro and the package).

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 1, 2017, 9:13pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/8 "2017-05-01T21:13:04Z")

</div>

(ps: 0.5 is supported now, sorry about that!)

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [May 1, 2017, 9:13pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/9 "2017-05-01T21:13:59Z")

</div>

It is also worth pointing out that this syntax has been around in C# for a very long time. Not clear to me whether Python or C# introduced it, but in any case, that would also support a name that doesn’t have “Python” in it.

I actually really wish julia had this build in, like Python and C#…

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 1, 2017, 9:16pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/10 "2017-05-01T21:16:26Z")

</div>

Oh, fair enough! Yeah, seems especially narrow to call it Py-anything then doesn’t it.

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 1, 2017, 9:22pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/11 "2017-05-01T21:22:03Z")

</div>

I have to agree about wishing this idiom was in julia natively.

```julia
function f(x)
...
   yield y
...
end

```

is much more clean than the alternative:

```julia
function f(x)
    function temp(c)
    ...
        push!(c, y)
    ...
    end
    return Channel(c -> temp(c))
end

```

Though, that might just be my lack of creativity…

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [May 1, 2017, 10:01pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/12 "2017-05-01T22:01:51Z")

</div>

FYI, `Channel(c -> temp(c))` can be simplified to just `Channel(temp)`. Even better, with do-syntax:

```julia
function f(x)
    return Channel() do c
        ...
        push!(c, y)
        ...
    end
end

```

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 2, 2017, 1:25am UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/13 "2017-05-02T01:25:25Z")

</div>

@cstjean nice!

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [May 2, 2017, 4:33am UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/14 "2017-05-02T04:33:00Z")

</div>

And even better (in my opinion), combining it with short-form function syntax,

```julia
f(x) = Channel() do c
    ...
    push!(c, y)
    ...
end

```

Effectively a macro-free way of defining a python-like generator.

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 2, 2017, 4:59am UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/15 "2017-05-02T04:59:57Z")

</div>

Yeah, I think I have to agree. I’ve always had reservation about the do-syntax because I find it reads a little weird, but this statement as really great. Its very clear that you’re filling up a channel with values.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [May 2, 2017, 5:00am UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/16 "2017-05-02T05:00:56Z")

</div>

Ah, I might have misunderstood the Python use of `yield`. I would like the C# version of `yield`, which simply is a shortcut to implementing an iterator, without the overhead of `Channel` etc. That is what I would really like to see in julia.

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 2, 2017, 5:10am UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/17 "2017-05-02T05:10:07Z")

</div>

You’re partially correct about the python `yield` statement from what I understand. They construct a `generator` which is a type of iterator, but I think that iterator is implemented with a type of coroutine, similar to the julia `Task`. The `@pygen` macro implements the closest julia equivalent I can find: a `Task` that communicates over a `Channel` in v0.6, or previous to v0.6 a `Task` that communicates via `produce()` and `consume()`.

Now, if there was a way to compile a function with yield statements into a nice high performance iterator (sounds like c# does this?) that would be amazing… A brief play with these Channels shows they’re no competition for an iterator when it comes to performance.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [May 2, 2017, 12:00pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/18 "2017-05-02T12:00:03Z")

</div>

> [@nsmith](#):
>
> but I think that iterator is implemented with a type of coroutine, similar to the julia Task.

No I don’t think it’s similar to Task.

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 2, 2017, 2:00pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/19 "2017-05-02T14:00:19Z")

</div>

You can see what they actually are [here](http://aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html) under the section “How Python Generators Work”. I couldn’t find much detail on the Julia’s `Task` implementation but looking at the [source](https://github.com/JuliaLang/julia/blob/master/src/task.c) is smells pretty similar (a stack frame + pointer to the instruction we’re at currently etc).

I am kind of curious about what C# is doing though, just because some calls itself an “iterator” doesn’t preclude the same co-routine stuff to be happening in the background I suppose.

---

<div class="post-metadata">

**Author:** ![nsmith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsmith/32/979_2.png) [@nsmith](https://discourse.julialang.org/u/nsmith)\
**Post date:** [May 2, 2017, 2:11pm UTC](https://discourse.julialang.org/t/pygen-python-style-generators/3451/20 "2017-05-02T14:11:57Z")

</div>

It seems that C# got away without a stack. They just expand the whole thing to in the callers stack as an object carrying the state and a bunch of goto statements to hop from where the iterator consumed and back again. Article [here](https://blogs.msdn.microsoft.com/oldnewthing/20080812-00/?p=21273).

[Next page](https://discourse.julialang.org/t/pygen-python-style-generators/3451.md?page=2)