# Defining a function with Iterators.filter - slower performance?

**URL:** <https://discourse.julialang.org/t/defining-a-function-with-iterators-filter-slower-performance/66451>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [August 16, 2021, 12:05am UTC](https://discourse.julialang.org/t/defining-a-function-with-iterators-filter-slower-performance/66451 "2021-08-16T00:05:01Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![pbaumgartner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbaumgartner/32/28214_2.png) [@pbaumgartner](https://discourse.julialang.org/u/pbaumgartner)\
**Post date:** [August 16, 2021, 12:05am UTC](https://discourse.julialang.org/t/defining-a-function-with-iterators-filter-slower-performance/66451/1 "2021-08-16T00:05:01Z")

</div>

Hello! Fairly new to Julia, I’m having an unexpected performance issue in defining a function that returns a `Iterators.filter` object.

I have a simulation that frequently requires me filtering all of my agents on some property, so I thought it would be DRY to capture that in a function – `conditional_property_iterator` below. However, when I attempt to actually run that function, it’s making almost twice as many allocations as the raw iterator and taking almost 2x as long.

What am I missing here that’s causing this performance issue?

I’ve got an MRE below, any help would be greatly appreciated. Note the adding `rand()` in the example below is just to do _something_ inside the loop, rather than the actual logic I have in my simulation.

```julia
using BenchmarkTools

struct Thing
    somevalue::Int
end

things = [Thing(rand(1:5)) for _ = 1:100000]

threes_getprop = Iterators.filter(a -> getproperty(a, :somevalue) === 3, things)
threes_prop = Iterators.filter(a -> a.somevalue === 3, things)

conditional_property_iterator(iterable, property::Symbol, value) = Iterators.filter(a -> getproperty(a, property) === value, iterable)
threes_cond = conditional_property_iterator(things, :somevalue, 3)

function test_threes_getproperty()
    for i in threes_getprop
        i.somevalue + rand(Int)
    end
end

function test_threes_prop()
    for i in threes_prop
        i.somevalue + rand(Int)
    end
end

function test_threes_cond()
    for i in threes_cond
        i.somevalue + rand(Int)
    end
end

```

```julia
julia> @btime test_threes_getproperty()
  2.834 ms (119301 allocations: 2.12 MiB)

julia> @btime test_threes_prop()
  2.750 ms (119301 allocations: 2.12 MiB)

julia> @btime test_threes_cond()
  4.491 ms (219301 allocations: 3.65 MiB)

```

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [August 16, 2021, 12:16am UTC](https://discourse.julialang.org/t/defining-a-function-with-iterators-filter-slower-performance/66451/2 "2021-08-16T00:16:57Z")

</div>

I am not entirely sure what causes what you are seeing. But there are many global variables in quite tight loops, so I’m not sure that I would trust any of these numbers very much.

What may capture what you’re asking, without globals, is this comparison:

```julia
julia> @btime sum(1 for a in $things if a.somevalue === 3) # uses Iterators.filter, hard-coded .somevalue
  107.542 μs (0 allocations: 0 bytes)
19792

julia> sum_prop(field, things) = sum(1 for a in things if getproperty(a, field) === 3);

julia> @btime sum_prop(:somevalue, $things) # uses Iterators.filter, field name as an argument
  107.875 μs (0 allocations: 0 bytes)
19792

julia> @btime sum_prop($(Ref(:somevalue))[], $things) # trying to defeat constant propagation
  1.686 ms (100000 allocations: 1.53 MiB)
19792

```

Even faster, here, is to avoid Iterators.filter entirely. If the loop predictably moves forwards, instead of having to check every element, it can often be optimised better. This may even pay the cost of allocating an intermediate array to store things:

```julia
julia> @btime count(a.somevalue === 3 for a in $things) # iterates all elements, no Iterators.Filter
  15.541 μs (0 allocations: 0 bytes)
19792

julia> @btime sum(map(a -> a.somevalue == 3, $things)) # makes an Array, could use `filter`
  43.583 μs (2 allocations: 97.73 KiB)
19792

```

Above I’m using the syntax for Iterators.filter, instead of writing that out:

```julia
julia> (x^2 for x in 1:10 if x>5)
Base.Generator{Base.Iterators.Filter{var"#60#62", UnitRange{Int64}}, var"#59#61"}(var"#59#61"(), Base.Iterators.Filter{var"#60#62", UnitRange{Int64}}(var"#60#62"(), 1:10))

julia> [x^2 for x in 1:10 if x>5] # collect(ans)
5-element Vector{Int64}:
  36
  49
  64
  81
 100

```

---

<div class="post-metadata">

**Author:** ![pbaumgartner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbaumgartner/32/28214_2.png) [@pbaumgartner](https://discourse.julialang.org/u/pbaumgartner)\
**Post date:** [August 16, 2021, 2:15pm UTC](https://discourse.julialang.org/t/defining-a-function-with-iterators-filter-slower-performance/66451/3 "2021-08-16T14:15:28Z")

</div>

Thanks for the response! I wasn’t aware of the alternative syntax for `Iterators.filter`, which is super h elpful.

I appreciate all the alternatives you provided, but I’m still not understanding why my function version is slower. In my application I’m not accumulating or doing any exclusively forward loops. I’m still iterating through each individual item and then running some complex logic for each item, and it doesn’t reduce to anything at the end. For more context, I am also using [FLoops.jl](https://github.com/JuliaFolds/FLoops.jl) to speed up these loops, so I am still looking for something that would allow me to iterate through items in a `for` loop.

---

<div class="post-metadata">

**Author:** ![pbaumgartner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbaumgartner/32/28214_2.png) [@pbaumgartner](https://discourse.julialang.org/u/pbaumgartner)\
**Post date:** [August 18, 2021, 12:06am UTC](https://discourse.julialang.org/t/defining-a-function-with-iterators-filter-slower-performance/66451/4 "2021-08-18T00:06:52Z")

</div>

I overlooked your comment about the global variables and rewrote the testing function. Still seeing more allocations/longer for the last one.

```julia
using BenchmarkTools

struct Thing
    somevalue::Int
end

things = [Thing(rand(1:5)) for _ = 1:100000]

threes_getprop = Iterators.filter(a -> getproperty(a, :somevalue) === 3, things)
threes_prop = Iterators.filter(a -> a.somevalue === 3, things)

conditional_property_iterator(iterable, property::Symbol, value) =
    Iterators.filter(a -> getproperty(a, property) === value, iterable)
threes_cond = conditional_property_iterator(things, :somevalue, 3)

function test_threes(iter)
    for i in iter
        i.somevalue + rand(Int)
    end
end

@btime test_threes(threes_getprop)
@btime test_threes(threes_prop)
@btime test_threes(threes_cond)

```

```julia
julia scratch/mre.jl
  293.765 μs (0 allocations: 0 bytes)
  289.539 μs (0 allocations: 0 bytes)
  1.961 ms (100000 allocations: 1.53 MiB)

```

I’m sure how to look into those allocations more from what I have here.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [August 18, 2021, 2:28am UTC](https://discourse.julialang.org/t/defining-a-function-with-iterators-filter-slower-performance/66451/5 "2021-08-18T02:28:16Z")

</div>

Seems like the culprit is the capturing of the property value by the closure argument of `filter`. This exhibits the same behaviour:

```julia
function conditional_property_iterator2(iterable, value)
    property = :somevalue
    Iterators.filter(a -> getproperty(a, property) === value, iterable)
end

```