Lazy Iteration

General Usage
1

I have defined a certain type that can hold arbitrarily many values, and I have already implemented an iterate() method for it. However, it seems to me that this method allocates all of the iterators elements in memory since it made me implement a Base.length(MyType) method in order for it to work.

Is there a way in Julia to implement an Iterator that does not allocate its objects in memory until they are needed? Given that I already have an iterator that calculates elements based on the state of it.

2

If length can be easily computed it allows various optimizations (like preallocation) when the iterator is materialized (like with collect). But it will not hold everything in memory when just iterating over it. Each item is retrieved sequentially with a call to iterate.

3 Likes
3

I tried printing my type in a for loop and it appears this is correct. However, do you know if the cycle and drop methods cause the iterator to materialize? (This is what caused me to ask the question in the first place, as it was the methods I was using)

4

They do not, they are “wrapper” iterators, implemented by calling iterate on the iterator they wrap.

5

I’ve been using Base.SizeUnknown() for iterators where it’s just as expensive to calculate length as to iterate over the entire collection.

Base.IteratorSize(::Type{OneRing}) = Base.SizeUnknown()
3 Likes