# An experiment for Python Style (but Unidirectional) Generators for Julia

**URL:** <https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451>\
**Category:** Internals & Design\
**Created:** [February 14, 2022, 6:36pm UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451 "2022-02-14T18:36:11Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![complyue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/complyue/32/33802_2.png) [@complyue](https://discourse.julialang.org/u/complyue)\
**Post date:** [February 14, 2022, 6:36pm UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/1 "2022-02-14T18:36:11Z")

</div>

I find it’s fairly trivial to implement Python style generator functions for Julia, though only unidirectional stream production.

I use `yieldto` but doc says “Its use is discouraged.”, how idiomatic my approach is? I’m fairly new to Julia, eagerly learning.

Surface syntax:

```nohighlight
julia> @generator function g(n::Int)
         m = n * 3
         @yield m
         m += 7
         @yield m
         m -= 9
         @yield m
       end
g (generic function with 1 method)

julia> for i in g(5)
         println(i)
       end
15
22
13

julia> collect(g(5))
3-element Vector{Int64}:
 15
 22
 13

julia>

```

Implementation source code here:  
[https://github.com/complyue/PyStyleButUnidirGenerators.jl/blob/main/src/PyStyleButUnidirGenerators.jl](https://github.com/complyue/PyStyleButUnidirGenerators.jl/blob/main/src/PyStyleButUnidirGenerators.jl)

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [February 14, 2022, 7:59pm UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/2 "2022-02-14T19:59:02Z")

</div>

I think the style developed in [FGenerators.jl](https://github.com/JuliaFolds/FGenerators.jl) combined with [FLoops.jl](https://github.com/JuliaFolds/FLoops.jl) is a more performant and well founded way to go about this, rather than spawning tasks and trying to properly manage the yielding. But it relies heavily on transducers, so that is a lot of infrastructure to digest and learn.

Here’s an example where I believe your use of `yieldto` fails:

```julia
julia> using .PyStyleButUnidirGenerators

julia> @generator function organpipe(n::Integer)
           i = 0
           while i != n
               i += 1
               @yield i
           end
           while true
               i -= 1
               i == 0 && return
               @yield i
           end
       end;

julia> collect(organpipe(2))

```

I left this on my computer for about 5 minutes and it never completed. I assumer there is some sort of task deadlock. I think if you want to do this with tasks, you’re better off making a channel, and then `put!`ing data into that channel at each `yield`, rather than trying to manage the task switching yourself.

Here’s how `FGenerators.jl` performs with that organpipe:

```julia
julia> using FGenerators

julia> @fgenerator function organpipe(n::Integer)
           i = 0
           while i != n
               i += 1
               @yield i
           end
           while true
               i -= 1
               i == 0 && return
               @yield i
           end
       end;

julia> let n = Ref(100)
           @btime sum(organpipe($n[]))
           @btime collect(organpipe($n[]))
       end;
  13.867 ns (0 allocations: 0 bytes)
  1.456 μs (204 allocations: 13.80 KiB)

```

---

<div class="post-metadata">

**Author:** ![complyue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/complyue/32/33802_2.png) [@complyue](https://discourse.julialang.org/u/complyue)\
**Post date:** [February 15, 2022, 10:32am UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/3 "2022-02-15T10:32:00Z")

</div>

Nice to see so many approaches have been attempted (the [See also](https://github.com/JuliaFolds/GeneratorsX.jl#see-also) section of [GeneratorsX.jl](https://github.com/JuliaFolds/GeneratorsX.jl) lists many of them)! Also parallelism and solo performance are tackled well.

* * *

I found a bug with your test case, it’s not related to the usage of `yieldto`, but inappropriate handling of early-return. It’s fixed as [https://github.com/complyue/PyStyleButUnidirGenerators.jl/commit/a425289c95c79e77f443650f19a53ac4bdd1ae7c](https://github.com/complyue/PyStyleButUnidirGenerators.jl/commit/a425289c95c79e77f443650f19a53ac4bdd1ae7c)

It works after the fix:

```nohighlight
julia> using PyStyleButUnidirGenerators

julia> @generator function organpipe(n::Integer)
         i = 0
         while i != n
           i += 1
           @yield i
         end
         while true
           i -= 1
           i == 0 && return
           @yield i
         end
       end
organpipe (generic function with 1 method)

julia> collect(organpipe(2))

3-element Vector{Int64}:
 1
 2
 1

julia> 

```

* * *

For cases not concerning HPC (parallelism included), I still favor my cooperative-scheduling based implementation over `Channel` based ones (all in the low-performance camp), I see `Channel` comm is somewhat _over demanding_ compared to cooperative-scheduling. I.e. as like the Python generators, the producing and consuming are interleaved and always _synchronous_, with cooperative-scheduling, no cost of **mutex** or **memory barrier/fence** would incur, unless the procedures perform async comm meanwhile (which can be nicely opted straight forward).

This somehow aligns to C++'s philosophy that “Pay only for what you use”.

* * *

I’m surprised none of those attempts addresses Python generator’s backward communication facility, i.e. `.send()` method of a generator call instance. Or I missed something?

Though that’s of limited usefulness, and Julia’s `yieldto()` works almost the same - just record your caller `Task` as a yield target (my impl. just leveraged that).

I feel Julia’s macro system can give even better ergonomics in use cases of that, but don’t have a particular one in my head atm.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [February 15, 2022, 11:47am UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/4 "2022-02-15T11:47:20Z")

</div>

Julia’s `Task` is very different from “stackless” coroutines/generators of Python/C++/Rust/etc. In Julia, a genuine call stack is allocated when starting a `Task`. Furthermore, the optimizer does not reason about the code across multiple tasks. As such, `yieldto`-based implementation of the generator (at least currently) will have significant non-optimizable overheads.

Coroutines and generators are interesting programming devices and it’d be nice to see more uses in Julia. For example, it’s useful for writing composable parsing tools. But I think there’re not many uses of `yieldto` in the Julia ecosystem because of the overhead of `yieldto` and people tend to be crazy about performance.

---

<div class="post-metadata">

**Author:** ![complyue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/complyue/32/33802_2.png) [@complyue](https://discourse.julialang.org/u/complyue)\
**Post date:** [February 15, 2022, 1:23pm UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/5 "2022-02-15T13:23:54Z")

</div>

Nice to know!

I guess each `Task` has its own dedicated native stack, so it runs at native machine speed until the next `yieldto` or other `yield` points, this is pretty lovable as with Julia 🙂

And with majority of high-performance parts well optimized, I suggest once you need to `yield`, there usually be some not-quite-performance-friendly situations to handle, e.g. needs to expand the capacity of some buffer or series storage backed by some database, then some slowdown could be relative forgivable in such cases, as long as full machine speed can be frictionlessly resumed after the situation settled. I see Julia being superior in achieving such a goal.

In handling low frequent situations, especially complex (w.r.t. business rules etc.) ones, ergonomics might be more valuable than run-speed, as code maintenance and other software-engineering endeavors could be rather more costing, over marginally less-time-to-run as in sense of business values for a return.

* * *

I’ve been longing to code in Julia since years ago, but can only embark until recently. Loving Julia as always, but tbh there’s a little pity that I feel it kinda being a DSL for high-machine-performance programming domain, I’d expect `esc`, `gensym`, `fieldcount` and alikes to only live in `Meta` and require explicit citation to use, but they are exposed from `Base`. Personally I’d regard them pollution to the conception space of business domain, when I deliver business functionalities to analytic team of my org, if as in Julia API. Analysts should learn Julia for sure, but bloated interfaces and tools with performance-optimization focus are, not welcoming to citizen developers consisting of many non-programmers but deeply involved in the computer-powered business.

I’m experimenting with composable grammar (syntax+semantics+pragmatics) components based on Julia, and feeling delighted. 😃 Hopefully I can share something in the near future.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [February 15, 2022, 3:44pm UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/6 "2022-02-15T15:44:02Z")

</div>

Have you seen [https://github.com/BenLauwens/ResumableFunctions.jl](https://github.com/BenLauwens/ResumableFunctions.jl)? AFAIK it’s the spiritually closest equivalent to Python-style generators.

---

<div class="post-metadata">

**Author:** ![complyue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/complyue/32/33802_2.png) [@complyue](https://discourse.julialang.org/u/complyue)\
**Post date:** [February 15, 2022, 4:46pm UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/7 "2022-02-15T16:46:13Z")

</div>

Yes, I later went over it indirectly through links in @Mason 's reply.

I’m aware its `Channel` based, and seemingly neither support communication back into the generator from outside. Also the limitations listed in its [Caveats](https://github.com/BenLauwens/ResumableFunctions.jl#caveats) section of readme:

- In a `try` block only top level `@yield` statements are allowed.
- In a `finally` block a `@yield` statement is not allowed.
- An anonymous function can not contain a `@yield` statement.

None of those exists in my cooperative-scheduling implementation, I suspect.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [February 15, 2022, 4:57pm UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/8 "2022-02-15T16:57:09Z")

</div>

I’m pretty sure it’s not channel-based? The readme explicitly calls out channels as slower for this use case and even benchmarks against them. The same benchmarks also include a [task-based implementation](https://github.com/BenLauwens/ResumableFunctions.jl/blob/v0.6.1/benchmark/benchmarks.jl#L53-L91). WRT calling back into the generator, that’s supported via [Manual · ResumableFunctions](https://benlauwens.github.io/ResumableFunctions.jl/dev/manual/#Two-way-communication).

---

<div class="post-metadata">

**Author:** ![complyue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/complyue/32/33802_2.png) [@complyue](https://discourse.julialang.org/u/complyue)\
**Post date:** [February 16, 2022, 8:47am UTC](https://discourse.julialang.org/t/an-experiment-for-python-style-but-unidirectional-generators-for-julia/76451/9 "2022-02-16T08:47:47Z")

</div>

Ah, I was careless and misread its readme.

The benchmarks have good coverage, nice insights!

* * *

The two-way communication supported there is quite like Python’s `.send()` semantics too.

But with Julia’s extra `macro`-fu, I’d imagine something more ergonomic could get en-sugar-ed, like:

```julia
@generator function questionnaire()
  name = @yield "Who are your?"
  from = @yield "Where are you from?"
  age = @yield "How old are you?"

  return "Hello, $name from $(from)!\n" *
         "You are born in $(year(today())-age)."
end

```

Then:

```julia
@consume questionnaire() do q
  if q === "Who are your?"
    @feedback "John"
  elseif q === "Where are you from?"
    @feedback "Chicago"
  elseif q === "How old are you?"
    @feedback 21
  end
end

```

Which evaluates to `"Hello John from Chicago!\nYou are born in 2001."`

This use case is too fictional to be useful in real world cases, but seems interesting and maybe someone (including future me) can find a really useful scenario.