Why is iterating over an Iterators.drop slower than iterating over the parent?

General Usage
,
1

Two ways to compute the last element of an iterator itr are:

  1. foldl((_, y) -> y, itr)
  2. first(Iterators.drop(itr, length(itr) - 1))

however, the second seems much slower than the first. I wonder why this is the case?

For example:

julia> itr = Iterators.product(1:3000, 2:3000);

julia> @btime foldl((_, y) -> y, $itr)
  6.440 ms (0 allocations: 0 bytes)
(3000, 3000)

julia> @btime first(Iterators.drop($it, length($it) - 1))
  9.534 ms (0 allocations: 0 bytes)
(3000, 3000)

Iterating over an Iterators.Drop effectively relies on iterating over the parent, so I am not sure why there’s this difference?

2

I can replicate this, but don’t have an explanation. Note that for some other iterables (eg UnitRange), the timings are reversed.

3

Just in case, @stevengj’ s solution posted here for computing the last element of an iterator is lightning-fast:

julia> @btime first(Iterators.reverse($itr))
  6.600 ns (0 allocations: 0 bytes)
(3000, 3000)
1 Like
4

This is applicable for iterators that implemented reverse, otherwise it is not faster than traversing the iterator.

2 Likes