# \[ANN\] FGenerators.jl: Python-like generator syntax for high-performance iteration

**URL:** <https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402>\
**Category:** Package Announcements\
**Tags:** loops\
**Created:** [August 23, 2020, 4:31am UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402 "2020-08-23T04:31:39Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 23, 2020, 4:31am UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402/1 "2020-08-23T04:31:39Z")

</div>

[FGenerators.jl](https://github.com/JuliaFolds/FGenerators.jl) is a package for defining [Transducers.jl](https://github.com/JuliaFolds/Transducers.jl)-compatible iteration protocol (aka extended `foldl`) with a simple Python-like `@yield`-based syntax. Quoting FGenerators.jl README, here are a few examples for creating ad-hoc “generators”:

> ```julia
> julia> using FGenerators
> 
> julia> @fgenerator function generate123()
> @yield 1
> @yield 2
> @yield 3
> end;
> 
> julia> collect(generate123())
> 3-element Array{Int64,1}:
> 1
> 2
> 3
> 
> julia> sum(generate123())
> 6
> 
> julia> @fgenerator function organpipe(n::Integer)
> i = 0
> while i != n
> i += 1
> @yield i
> end
> while true
> i -= 1
> i == 0 && return
> @yield i
> end
> end;
> 
> julia> collect(organpipe(3))
> 5-element Array{Int64,1}:
> 1
> 2
> 3
> 2
> 1
> 
> julia> @fgenerator function organpipe2(n)
> @yieldfrom 1:n
> @yieldfrom n-1:-1:1
> end;
> 
> julia> collect(organpipe2(2))
> 3-element Array{Int64,1}:
> 1
> 2
> 1
> 
> ```

You can use [FLoops.jl](https://github.com/JuliaFolds/FLoops.jl) (see also: [ANN: Parallel `for` loops in FLoops.jl with composable and extensible fold-based API](https://discourse.julialang.org/t/ann-parallel-for-loops-in-floops-jl-with-composable-and-extensible-fold-based-api/45390)) to iterate over the items yielded from the “generator”

> ```julia
> julia> using FLoops
> 
> julia> @floop for x in generate123()
> @show x
> end
> x = 1
> x = 2
> x = 3
> 
> ```

That is to say, FGenerators.jl is a framework to write loops _except_ the loop body and FLoops.jl is a framework to fill out the loop body. Transducers.jl’s core provides the interface between them.

FGenerators.jl is a spin-off of [GeneratorsX.jl](https://github.com/JuliaFolds/GeneratorsX.jl) that not only defines `foldl` but also `iterate` using IRTools.jl. I am releasing the easier half (=`foldl`) as FGenerators.jl since IRTools.jl is a big dependency and inherently not the most robust package due to the reliance on some `julia` internals.

## Why not `iterate`?

Note that `@fgenerator` does not create the generator in the sense of `Base.Generator`. That is to say, it does not define `iterate`. The reason behind using a protocol other than `iterate` is that you have more control over the execution strategy of the loop (it is also [discussed in GeneratorsX.jl README](https://github.com/JuliaFolds/GeneratorsX.jl#why)). For example, here is an [implementation](https://github.com/JuliaFolds/LoopRecipes.jl/blob/c1c33c0cc090ee9c7345e423e3b40080d6c207a7/src/simd.jl#L34-L55) of `LoopRecipes.simdeachindex` ([documentation](https://juliafolds.github.io/LoopRecipes.jl/dev/#LoopRecipes.simdeachindex-Tuple%7BAny%7D)) that yields [`SIMD.VecRange`](https://github.com/eschnett/SIMD.jl)s in the main part of the loop and normal integers in the remainder

```julia
simdeachindex(xs) = simdeachindex(Val{pick_vector_width(eltype(xs))}(), xs)
simdeachindex(width::Val, xs) = SIMDEachIndex(width, firstindex(xs), lastindex(xs))

@fgenerator(foldable::SIMDEachIndex) do
    W = valof(foldable.width)
    i = foldable.firstindex
    n = foldable.lastindex - W
    lane = VecRange{W}(0)
    if i <= n
        @yield lane + i
        i += W
        while i <= n
            @yield lane + i
            i += W
        end
    end
    if i <= foldable.lastindex
        @yield i
        i += 1
        while i <= foldable.lastindex
            @yield i
            i += 1
        end
    end
end

```

This can be used to abstract out the handling of the main and remainder loop

```julia
julia> using LoopRecipes, FLoops

julia> @floop for i in simdeachindex(ones(10))
           @show i
       end;
i = VecRange{4}(1)
i = VecRange{4}(5)
i = 9
i = 10

```

For more practical example of this, see sparse-dense dot product (`simddot(xs::SparseVector, ys)`) example in [`LoopRecipes.simdstored` documentation](https://juliafolds.github.io/LoopRecipes.jl/dev/#LoopRecipes.simdstored-Tuple%7BAny%7D).

Unfortunately, it seems like Julia’s optimizer do not optimize the loops using something like `simdeachindex` if it were implemented using `iterate`. Such loops have interesting progressions in the type of accumulator and element. It’s probably that there is not enough mechanism in the compiler to extract out this structure (I think the gap between `iterate` and `foldl` can be filled if [someone like Keno takes time to write more abstract interpretation code](https://github.com/JuliaLang/julia/pull/36524)). But, even if the compiler becomes wise enough, I’d argue that the equivalent `iterate` definition is rather hard to come up because you need to construct the state machine by hand:

```julia
@inline function Base.iterate(foldable::SIMDEachIndex)
    W = valof(foldable.width)
    i = foldable.firstindex
    n = foldable.lastindex - W
    return iterate(foldable, (i, n))
end

@inline function Base.iterate(foldable::SIMDEachIndex, (i, n)::Tuple{Int,Int})
    W = valof(foldable.width)
    if i <= n
        return (VecRange{W}(i), (i + W, n))
    else
        return iterate(foldable, i)
    end
end

@inline function Base.iterate(foldable::SIMDEachIndex, i::Int)
    i > foldable.lastindex && return nothing
    return (i, i + 1)
end

```

## Other `yield` syntax packages

For comparisons to other packages that provides “`yield` syntax” such as [Continuables.jl](https://github.com/schlichtanders/Continuables.jl) ([JuliaCon 2020 talk](https://www.youtube.com/watch?v=HkOufyJBN24)), [ResumableFunctions.jl](https://github.com/BenLauwens/ResumableFunctions.jl), [PyGen](https://discourse.julialang.org/t/pygen-python-style-generators/3451), see the discussion in [GeneratorsX.jl README](https://github.com/JuliaFolds/GeneratorsX.jl#see-also).

---

<div class="post-metadata">

**Author:** ![djholiver](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/djholiver/32/50470_2.png) [@djholiver](https://discourse.julialang.org/u/djholiver)\
**Post date:** [February 18, 2022, 8:28pm UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402/2 "2022-02-18T20:28:27Z")

</div>

Hi,

This is such a fantastic package - thank you for creating it.

I was successfully using it for [some time](https://discourse.julialang.org/t/composing-collection-of-iterators-that-accept-iterators-and-collecting-the-result/66082), including earlier today, until a recent update to 1.7.2 and now receive the following issues:

> **errors**
>
> `ERROR: LoadError: Method Error: no method matching length(::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}) Closest candidates are: length(!Matched::Union{Base.KeySet, Base.ValueIterator}) at abstractdict.jl:58 length(!Matched::Union{Tables.AbstractColumns, Tables.AbstractRow}) at C:\Users\me\.julia\packages\Tables\M26tI\src\Tables.jl:175 length(!Matched::Union{DataStructures.OrderedRobinDict, DataStructures.RobinDict}) at C:\Users\me\.julia\packages\DataStructures\vSp4s\src\ordered_robin_dict.jl:86 ... Stacktrace: [1] amount(xs::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}) @ SplittablesBase.Implementations ~\.julia\packages\SplittablesBase\gpREK\src\implementations.jl:65 [2] transduce_assoc(xform::Transducers.IdentityTransducer, step::Transducers.AdHocRF{Main.GetMetricService.QueryResolver.var"##oninit_function#361#81", typeof(identity), Main.GetMetricService.QueryResolver.var"##reducing_function#362#82"{Vector{String}, Arrow.Table, Dict{String, Vector{Tuple{Int64, Float64}}}, Arrow.Primitive{Union{Missing, Int64}, Vector{Int64}}}, typeof(identity), typeof(identity), Main.GetMetricService.QueryResolver.var"##combine_function#363#83"}, init::Transducers.InitOf{Transducers.DefaultInitOf}, coll0::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}; simd::Val{false}, basesize::Nothing, stoppable::Nothing, nestlevel::Nothing) @ Transducers ~\.julia\packages\Transducers\CnpYX\src\reduce.jl:108 [3] transduce(xf::Transducers.IdentityTransducer, rf::Function, init::Transducers.InitOf{Transducers.DefaultInitOf}, coll::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}, exc::ThreadedEx{NamedTuple{(:simd,), Tuple{Val{false}}}}) @ Transducers ~\.julia\packages\Transducers\CnpYX\src\executors.jl:152 [4] transduce(xf::Transducers.IdentityTransducer, rf::Function, init::Transducers.InitOf{Transducers.DefaultInitOf}, coll::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}, exc::PreferParallel{NamedTuple{(:simd,), Tuple{Val{false}}}}) @ Transducers ~\.julia\packages\Transducers\CnpYX\src\executors.jl:164 [5] _fold @ ~\.julia\packages\FLoops\3ZEuy\src\reduce.jl:851 [inlined] [6] _fold(rf::Transducers.AdHocRF{Main.GetMetricService.QueryResolver.var"##oninit_function#361#81", typeof(identity), Main.GetMetricService.QueryResolver.var"##reducing_function#362#82"{Vector{String}, Arrow.Table, Dict{String, Vector{Tuple{Int64, Float64}}}, Arrow.Primitive{Union{Missing, Int64}, Vector{Int64}}}, typeof(identity), typeof(identity), Main.GetMetricService.QueryResolver.var"##combine_function#363#83"}, coll::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}, #unused#::Nothing, simd::Val{false}) @ FLoops ~\.julia\packages\FLoops\3ZEuy\src\reduce.jl:849 [7] macro expansion @ ~\.julia\packages\FLoops\3ZEuy\src\reduce.jl:829 [inlined] [8] groupOverIndexSpace(column::String, colSet::Vector{String}, data::Arrow.Table, enumerable::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}) @ Main.GetMetricService.QueryResolver ~\source\repos\GetMetricEngineCore\src\QueryResolverIterable.jl:1983 [9] macro expansion @ ~\source\repos\GetMetricEngineCore\src\QueryResolverIterable.jl:1763 [inlined] [10] (::Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#323#53")( __rf__ #257::Transducers.BottomRF{Transducers.SideEffect{Main.GetMetricService.QueryResolver.var"#2#5"{Set{Any}}}}, __acc__ #258::Nothing, xs#324::NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.DataDominance, AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}, Int64, Int64}}) @ Main.GetMetricService.QueryResolver ~\.julia\packages\FGenerators\N3bhz\src\FGenerators.jl:290 [11] __foldl__ @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:1339 [inlined] [12] #transduce#140 @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:523 [inlined] [13] transduce @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:520 [inlined] [14] transduce(xform::Transducers.IdentityTransducer, f::Transducers.SideEffect{Main.GetMetricService.QueryResolver.var"#2#5"{Set{Any}}}, init::Nothing, coll::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#323#53", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.DataDominance, AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}, Int64, Int64}}}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Transducers ~\.julia\packages\Transducers\CnpYX\src\processes.jl:506 [15] transduce(xform::Transducers.IdentityTransducer, f::Transducers.SideEffect{Main.GetMetricService.QueryResolver.var"#2#5"{Set{Any}}}, init::Nothing, coll::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#323#53", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.DataDominance, AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}, Int64, Int64}}}) @ Transducers ~\.julia\packages\Transducers\CnpYX\src\processes.jl:505 [16] foreach(eff::Main.GetMetricService.QueryResolver.var"#2#5"{Set{Any}}, reducible::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#323#53", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.DataDominance, AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}, Int64, Int64}}}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Transducers ~\.julia\packages\Transducers\CnpYX\src\processes.jl:1128 [17] foreach(eff::Main.GetMetricService.QueryResolver.var"#2#5"{Set{Any}}, reducible::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#323#53", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.DataDominance, AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}}, Int64, Int64}}}) @ Transducers ~\.julia\packages\Transducers\CnpYX\src\processes.jl:1127 [18] macro expansion @ ~\source\repos\GetMetricEngineCore\src\QueryResolverIterable.jl:236 [inlined] [19] (::Main.GetMetricService.QueryResolver.var"##getData#foldl#257#3")( __rf__ #257::Transducers.BottomRF{Transducers.SideEffect{Main.GetMetricService.GetMetricBase.var"#1#2"{Vector{Main.GetMetricService.GetMetricModels.ComputeSetCollection}}}}, __acc__ #258::Nothing, xs#258::NamedTuple{(:queryModel,), Tuple{Main.GetMetricService.QueryModels.QueryDataModel}}) @ Main.GetMetricService.QueryResolver ~\.julia\packages\FGenerators\N3bhz\src\FGenerators.jl:290 [20] __foldl__ @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:1339 [inlined] [21] #transduce#140 @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:523 [inlined] [22] transduce @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:520 [inlined] [23] transduce(xform::Transducers.IdentityTransducer, f::Transducers.SideEffect{Main.GetMetricService.GetMetricBase.var"#1#2"{Vector{Main.GetMetricService.GetMetricModels.ComputeSetCollection}}}, init::Nothing, coll::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getData#foldl#257#3", NamedTuple{(:queryModel,), Tuple{Main.GetMetricService.QueryModels.QueryDataModel}}}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Transducers ~\.julia\packages\Transducers\CnpYX\src\processes.jl:506 [24] transduce @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:505 [inlined] [25] #foreach#149 @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:1128 [inlined] [26] foreach @ ~\.julia\packages\Transducers\CnpYX\src\processes.jl:1127 [inlined] [27] getMetric(requestID::String, queryResultsSet::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getData#foldl#257#3", NamedTuple{(:queryModel,), Tuple{Main.GetMetricService.QueryModels.QueryDataModel}}}, analyticsRequests::Vector{Main.GetMetricService.GetMetricRequests.GetMetricRequestModels.IAnalyticsRequest}, resultSetSchema::Vector{String}, analyticsColumns::Vector{String}, subsetSchema::Vector{String}) @ Main.GetMetricService.GetMetricBase ~\source\repos\GetMetricEngineCore\src\GetMetricBase.jl:64 [28] getMetric(getMetricRequest::Main.GetMetricService.GetMetricRequests.GetMetricRequestModels.GetMetricRequestEndPoint) @ Main.GetMetricService ~\source\repos\GetMetricEngineCore\src\GetMetricService.jl:90 [29] top-level scope @ show.jl:955 in expression starting at c:\Users\me\source\repos\GetMetricEngineCore\src\runtests.jl:287`

the manifest for the project is (I’ve omitted non - relevant packages since I exceeded the character limit originally):

> **Manifest**
>
> `
> 
> [[BangBang]]  
> deps = [“Compat”, “ConstructionBase”, “Future”, “InitialValues”, “LinearAlgebra”, “Requires”, “Setfield”, “Tables”, “ZygoteRules”]  
> git-tree-sha1 = “d648adb5e01b77358511fb95ea2e4d384109fac9”  
> uuid = “198e06fe-97b7-11e9-32a5-e1d131e6ad66”  
> version = “0.3.35”
> 
> [[FGenerators]]  
> deps = [“AbstractYieldMacros”, “ContextualMacros”, “FLoopsBase”, “MacroTools”, “Transducers”]  
> git-tree-sha1 = “dd2f057dbda6ad499e82cd89489865c14452a5d0”  
> uuid = “4fd0377b-cfdc-4941-97f4-8d7ddbb8981e”  
> version = “0.1.4”
> 
> [[FLoops]]  
> deps = [“BangBang”, “Compat”, “FLoopsBase”, “InitialValues”, “JuliaVariables”, “MLStyle”, “Serialization”, “Setfield”, “Transducers”]  
> git-tree-sha1 = “4391d3ed58db9dc5a9883b23a0578316b4798b1f”  
> uuid = “cc61a311-1640-44b5-9fba-1b764f453329”  
> version = “0.2.0”
> 
> [[FLoopsBase]]  
> deps = [“ContextVariablesX”]  
> git-tree-sha1 = “656f7a6859be8673bf1f35da5670246b923964f7”  
> uuid = “b9860ae5-e623-471e-878b-f6a53c775ea6”  
> version = “0.1.1”
> 
> [[SplittablesBase]]  
> deps = [“Setfield”, “Test”]  
> git-tree-sha1 = “39c9f91521de844bad65049efd4f9223e7ed43f9”  
> uuid = “171d559e-b47b-412a-8079-5efa626c420e”  
> version = “0.1.14”
> 
> [[Transducers]]  
> deps = [“Adapt”, “ArgCheck”, “BangBang”, “Baselet”, “CompositionsBase”, “DefineSingletons”, “Distributed”, “InitialValues”, “Logging”, “Markdown”, “MicroCollections”, “Requires”, “Setfield”, “SplittablesBase”, “Tables”]  
> git-tree-sha1 = “1cda71cc967e3ef78aa2593319f6c7379376f752”  
> uuid = “28d57a85-8fef-5791-bfe6-a80928e7c999”  
> version = “0.4.72”
> 
> `

with Julia version 1.6.0 (where it used to work but now fails after running Pkg.update()), 1.6.5 (where it didnt) and 1.7.2 (where it also didn’t).

Having attempted and failed to write a length / amount required for the @fgenerator, I’m reaching out in case this is a version issue or something easily resolved / reported by others. Google searches haven’t yielded anything to suggest this is well reported

I’m at a bit of a loss as this was a great case of “it just works” originally so hoping it’s something straightforward.

Regards,

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 18, 2022, 11:35pm UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402/3 "2022-02-18T23:35:53Z")

</div>

Pls mind the environment by using [this discourse tip](https://discourse.julialang.org/t/collapsible-code-sections-in-discourse/64803/2).

---

<div class="post-metadata">

**Author:** ![djholiver](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/djholiver/32/50470_2.png) [@djholiver](https://discourse.julialang.org/u/djholiver)\
**Post date:** [February 18, 2022, 11:49pm UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402/4 "2022-02-18T23:49:09Z")

</div>

Thanks for the tip!

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [February 19, 2022, 1:08am UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402/5 "2022-02-19T01:08:29Z")

</div>

Looking at the stacktrace, I’m going to guess that this is due to FLoops.jl’s change in 0.2: [https://github.com/JuliaFolds/FLoops.jl#update-notes](https://github.com/JuliaFolds/FLoops.jl#update-notes). By default `@floop for ...` is (mostly) always parallel. However, FGenerators.jl is not parallelizable by default.

You can use `SequentialEx()` as in `@floop SequentialEx() for ...` for FLoops.jl 0.1 behavior to avoid the error.

---

<div class="post-metadata">

**Author:** ![djholiver](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/djholiver/32/50470_2.png) [@djholiver](https://discourse.julialang.org/u/djholiver)\
**Post date:** [February 19, 2022, 1:21pm UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402/6 "2022-02-19T13:21:21Z")

</div>

Hi,

Thank you very much for reviewing the error and responding so quickly - that did indeed work in the sense that results are now being returned again!

It has made me think that to extend this to work with the parallel form might be a bit of a challenge given the error message. Would you advise writing this without the use of the @fgenerator macro and instead creating via the Transducer library proper? I did try a few combinations to try and mitigate the length error but no luck as this syntax from the error is a bit confusing:

```julia
length(::AdHocFoldable{Main.GetMetricService.QueryResolver.var"##getQueryIndexes#foldl#281#27", NamedTuple{(:tabularSource, :IQueryModel, :enumerable, :MinVal, :MaxVal), Tuple{Main.GetMetricService.QueryResolver.DataSetConverterModels.TabularSource, Main.GetMetricService.QueryModels.Between, Int64, Int64, Int64}}})

```

Anyway - thanks again - this package (and your Transducer ecosystem) is great and performant. I only had one challenge in composing multiple @fgenerators as per my link in the original post but worked around that via an initialiser of missing for the innermost loop:

```julia
function composeFGenerators(fgenerators)
initialIterator = missing

for fgenerator in fgenerators
   initialIterator = fgenerator(initialIterator)
end
return initialIterator
end

#used like this 

@floop for r in (composeFGenerators(fGenerators))
    @yield r
end  

```

Regards,

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [February 19, 2022, 2:55pm UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402/7 "2022-02-19T14:55:38Z")

</div>

See [https://github.com/JuliaFolds/FGenerators.jl#defining-parallelizable-collection](https://github.com/JuliaFolds/FGenerators.jl#defining-parallelizable-collection) for how to make `@fgenerator` parallelizable.

> [@djholiver](#):
>
> Would you advise writing this without the use of the @fgenerator macro and instead creating via the Transducer library proper?

`@fgenerator` is a very simple macro for generating (sequential) Transducers.jl API. There’s only a limited set of usage patterns where writing “raw” Transducers.jl API makes sense. For parallelization, you just have to add a couple of methods as mentioned in the link above.

---

<div class="post-metadata">

**Author:** ![djholiver](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/djholiver/32/50470_2.png) [@djholiver](https://discourse.julialang.org/u/djholiver)\
**Post date:** [February 19, 2022, 3:54pm UTC](https://discourse.julialang.org/t/ann-fgenerators-jl-python-like-generator-syntax-for-high-performance-iteration/45402/8 "2022-02-19T15:54:36Z")

</div>

Thanks - I’ll give it a go!

Regards