Difference between calling Iterators.rest on an array vs a range

New to Julia
,
1

I’m just curious why Iterators.rest yields different results depending on whether a range or an array is used. eg

df1 = DataFrame( :x=> 1:5)

To start a loop on the second iteration of a range I would use


collect(Iterators.rest(eachindex(df1.x),1)) # state = 1
4-element Vector{Int64}:
 2
 3
 4
 5

The example in the docs use an array, and to start on the second iteration of an array would be

collect(Iterators.rest(collect(eachindex(df1.x)),2)) # state = 2
4-element Vector{Int64}:
 2
 3
 4
 5

I was just wondering what accounts for the difference?

2

Minor but important point, you’re not looking at the range, you’re looking at its indices eachindex, which in this case is a Base.OneTo(5). But if you remove that method call, this discrepancy remains.

The main point here is that state is not an iteration count or the corresponding item. See how the first two items below (1,2) are the same for the range and the array, but the states are different? Although states indicate iteration progress, there’s no expectation we can intuit the state that produces the n-th iteration. The state can actually be anything, even strings*.

julia> iterate(1:5), iterate(1:5, 1)
((1, 1), (2, 2))

julia> iterate(collect(1:5)), iterate(collect(1:5), 2)
((1, 2), (2, 3))

*unintuitive strings as state

struct HelloWorld end

function Base.iterate(::HelloWorld, state=nothing)
  if state == "ZA"
    ("WORLD!", "STOP TIME")
  elseif state == "STOP TIME" # no more items
    nothing
  else # unexpected or no states start at Hello
    ("Hello ", "ZA")
  end
end

for i in HelloWorld() println(i) end        # Hello WORLD!
println(iterate(HelloWorld(), "whoops"))    # ("Hello ", "ZA")
println(iterate(HelloWorld(), "ZA"))        # ("WORLD!", "STOP TIME")
println(iterate(HelloWorld(), "STOP TIME")) # nothing
1 Like
3

As stated above, Iterators.rest takes a state as its second argument, which is usually an implementation detail of the iterator. It so happens that state=1 does not refer to the same iteration point for a range and a vector. Iterators.rest should mostly be used on the iterators that you define, since the iteration state is otherwise not usually part of any API.
If you are curious and want to look at the definition of iterate for ranges and for vectors, you can type @edit iterate(1:5) and @edit iterate([3, 5, 6]) in your REPL and look at the code. The full iteration protocol is detailed in the docs here and there.

In contrast, the user-facing Iterators.drop does not rely on the internals of the iteration protocol and can be used to skip the n first elements, as you would expect. With your examples, when dropping 1 element (i.e. starting at the second element of the iterator):

julia> collect(Iterators.drop(1:5, 1))
4-element Vector{Int64}:
 2
 3
 4
 5

julia> collect(Iterators.drop(collect(1:5), 1))
4-element Vector{Int64}:
 2
 3
 4
 5
3 Likes
4

Thank you both so much for clarifying that distinction in such a clear and understandable way. I don’t think I would have pick up on it otherwise. I just marked the first answer as there is currently no way to mark multiple solutions.

1 Like