PyGen - python style generators

package
announcement

#1

I am addicted to those yield statements in python. If you’re in the same boat checkout PyGen. I know you can achieve the same thing with Channels but its always felt a little clunky to me. Anyways, here is how it works:

julia> using PyGen

julia> @pygen """ 
function fibonacci()
    n, m = 0, 1
    while true
        yield m
        n, m = m, n + m
    end
end
"""
fibonacci (generic function with 1 method)

julia> for i in fibonacci()
           println(i)
           sleep(1)
       end
1
1
3
5
8
.
.
.

Enjoy :slight_smile:


#2

Cool! Have you considered writing a regular macro, instead of one operating on strings?

@pygen function fibonacci()
    n, m = 0, 1
    while true
        yield(m)
        n, m = m, n + m
    end
end

will be more robust, and won’t kill auto-indent/syntax highlighting. You might find MacroTools’ expression walking functions useful.


#3

Oh good call, I did it this way because the REPL would barf as soon as I wrote any yield x statements, but the yield(x) fixes that. MacroTools looks heavenly, thanks for the advice :slight_smile:


#4

If you don’t want to use yield(m) but have more of a statement feel, you can use @yield m. You don’t ever have to define a @yield macro if you make sure to handle and eliminate those @yield Exprs in the pygen macro.


#5

Thanks for the tips everyone! @pygen is even better than before :).

julia> using PyGen

julia> @pygen function pascal(n)
           i = 0
           while i <= n
               yield(i)
               i += 1
           end
       end
pascal (generic function with 1 method)

julia> n = 20;

julia> sum(pascal(n)) == n * (n + 1) / 2   # Booyah computers work
true 

I think I’m beginning to see the light on the whole macro business… Feels good to just add the features you want to a language.


#6

The name “PyGen” suggests that it is something to do with Python, whereas it isn’t really. Maybe something like “yieldgen”?


#7

I suppose as a package name it sort of clashes with the current trend of being a wrapper of a Python library or Python itself, but I don’t see why implementing a piece of syntax from Python is very different. Unfortunately, the vocabulary on the Julia side makes hard to talk about it without reference to Python.

@yield, @generator etc already have specific meaning in Julia… perhaps @pyyield because it supplies the Python type yield statement?

I do agree it has a troubled sort of name (both the macro and the package).


#8

(ps: 0.5 is supported now, sorry about that!)


#9

It is also worth pointing out that this syntax has been around in C# for a very long time. Not clear to me whether Python or C# introduced it, but in any case, that would also support a name that doesn’t have “Python” in it.

I actually really wish julia had this build in, like Python and C#…


#10

Oh, fair enough! Yeah, seems especially narrow to call it Py-anything then doesn’t it.


#11

I have to agree about wishing this idiom was in julia natively.

function f(x)
...
   yield y
...
end

is much more clean than the alternative:

function f(x)
    function temp(c)
    ...
        push!(c, y)
    ...
    end
    return Channel(c -> temp(c))
end

Though, that might just be my lack of creativity…


#12

FYI, Channel(c -> temp(c)) can be simplified to just Channel(temp). Even better, with do-syntax:

function f(x)
    return Channel() do c
        ...
        push!(c, y)
        ...
    end
end

#13

@cstjean nice!


#14

And even better (in my opinion), combining it with short-form function syntax,

f(x) = Channel() do c
    ...
    push!(c, y)
    ...
end

Effectively a macro-free way of defining a python-like generator.


Creating Generators
#15

Yeah, I think I have to agree. I’ve always had reservation about the do-syntax because I find it reads a little weird, but this statement as really great. Its very clear that you’re filling up a channel with values.


#16

Ah, I might have misunderstood the Python use of yield. I would like the C# version of yield, which simply is a shortcut to implementing an iterator, without the overhead of Channel etc. That is what I would really like to see in julia.


#17

You’re partially correct about the python yield statement from what I understand. They construct a generator which is a type of iterator, but I think that iterator is implemented with a type of coroutine, similar to the julia Task. The @pygen macro implements the closest julia equivalent I can find: a Task that communicates over a Channel in v0.6, or previous to v0.6 a Task that communicates via produce() and consume().

Now, if there was a way to compile a function with yield statements into a nice high performance iterator (sounds like c# does this?) that would be amazing… A brief play with these Channels shows they’re no competition for an iterator when it comes to performance.


From generator (with Channel) to iterator
#18

No I don’t think it’s similar to Task.


#19

You can see what they actually are here under the section “How Python Generators Work”. I couldn’t find much detail on the Julia’s Task implementation but looking at the source is smells pretty similar (a stack frame + pointer to the instruction we’re at currently etc).

I am kind of curious about what C# is doing though, just because some calls itself an “iterator” doesn’t preclude the same co-routine stuff to be happening in the background I suppose.


#20

It seems that C# got away without a stack. They just expand the whole thing to in the callers stack as an object carrying the state and a bunch of goto statements to hop from where the iterator consumed and back again. Article here.