Macro for Iterator

[leiteiro]

Given a mutable struct H and a function G(S::H) I want to
create this iterator:

Base.eltype(::Type{H}) = Int
Base.length(S::H) = S.a
Base.start(::H) = 1
Base.done(S::H, state) = state > S.a
Base.next(S::H, state) = (G(S), state + 1)

No problem so far. However I have to repeat this for many different
pairs (H, G) over and over again.

Since the docs say macros allow the programmer to generate and
include fragments of customized code I think they could help here.

How would such a macro look like?

For example: if H is the struct HelloState and the function G the
HelloGenerator below …

module HelloModule

mutable struct HelloState
    a::Int
    b::Int
end

function HelloGenerator(S::HelloState)
    S.b += 1; "Hello" * ^("!", S.b) * " "
end

… this macro would expand to the next 5 lines below:

Base.eltype(::Type{HelloState}) = Int
Base.length(S::HelloState) = S.a
Base.start(::HelloState) = 1
Base.done(S::HelloState, state) = state > S.a
Base.next(S::HelloState, state) = (HelloGenerator(S), state + 1)

Hello(count) = HelloState(count, 0)
for i in Hello(6) print(i) end
end # module

[Tamas_Papp]

I don’t think you need macros (unless I am missing something about your problem). Just make all types a subtype for some abstract type, and define the methods above for that type. Then it will work for all concrete subtypes.

[stevengj]

@leiteiro

Base.next(S::H, state) = (G(S), state + 1)

It seems a bit odd to define a different iterator for each function G. Are you sure you don’t just want an iterator that returns S and then use it with e.g. map or mapreduce to combine it with arbitrary functions G?

(Usually, next should not mutate the iterator object, only the state, though there are a few exceptions like eachline where the iterator has a side effect on the iterator object. It’s better to only modify the state, e.g. so you can have multiple/nested loops over the same iterator object.)

Iterator and generator concepts in Julia work a bit differently than they do in e.g. Python if you are coming from there.

[leiteiro]

Interesting idea. Let’s see if I have understood.
First I have to change the specification to:

Base.next(S::H, state) = (S.G(S), state + 1)

With a different example I than get:

abstract type Sequence end

Base.eltype(::Type{Sequence}) = Int
Base.length(S::Sequence) = S.count
Base.start(::Sequence) = 1
Base.done(S::Sequence, state) = state > S.count
Base.next(S::Sequence, state) = (S.f(S), state + 1)

mutable struct SqrState <: Sequence
	count::Int
	x::Int
	f::Function
end

function SqrGenerator(S::SqrState)
	S.x += 1; S.x * S.x
end

SqrSequence(count) = SqrState(count, 0, SqrGenerator)
for s in SqrSequence(6) print(s, " ") end

Suggestions for improvement?

[leiteiro]

I want to ‘hide’ the iterator and not to make it
explicite like it has to be with mapreduce.

I think of G as a ‘big’ state machine with states
in H which is only triggered by the small state
machine ‘iterator’.

It’s clear to me that I do not yet know how to
express this idea in a ‘Julian way’. Seems to be
much simpler to do in an object oriented setup.

[stevengj]

@leiteiro

I want to ‘hide’ the iterator and not to make it
explicite like it has to be with mapreduce.

Can you give a concrete example of what you would like to do?

[leiteiro]

“I want to ‘hide’ the iterator and not to make it
explicit like it has to be with mapreduce.”

It is intended to be used in a user API. The user needs
not to see the iteration details.

In fact the proposal of Tamas Papp does accomplish this.
See my example based on it: In the constructor

SqrSequence(count) = SqrState(count, 0, SqrGenerator) 

there is no explicit mention of an iterator.

I will mark Tamas proposal as a solution. What’s not nice
is that the constructor has to set the initial values explicitly,
i.e. that there is no way to give the struct default values
which would make it more elegant. This has been discussed
here but there seems to be no solution yet.

A different solution could be based on the produce/consume
pattern which has unfortunately been destroyed in Julia 0.6,
see this discussion.

[traktofon]

Hi @leiteiro,
note that in Julia it is not necessary (and arguably counterproductive) to put function fields into your objects. Your previous code proposal can be rewritten in a more “Julian” way as follows (I think this is what @Tamas_Papp had in mind):

abstract type Sequence end

Base.eltype(::Type{Sequence}) = Int
Base.length(S::Sequence) = S.count
Base.start(::Sequence) = 1
Base.done(S::Sequence, state) = state > S.count
Base.next(S::Sequence, state) = (G(S), state + 1)

mutable struct SqrState <: Sequence
   count::Int
   x::Int
end

function G(S::SqrState)
   S.x += 1; S.x * S.x
end

SqrSequence(count) = SqrState(count, 0)
for s in SqrSequence(6) print(s, " ") end

The function G, which is called in your next method, is specialized on the concrete iterator state type. To add another iterator, you then need to define another concrete state type, and add another method for G, as in the following example:

mutable struct ExpState <: Sequence
   count::Int
   x::Int
end

function G(S::ExpState)
   S.x += 1; exp(S.x)
end

ExpSequence(count) = ExpState(count, 0)
for s in ExpSequence(6) print(s, " ") end

[traktofon]

Going slightly off topic, the kind of generator/iterators discussed here can quite simply be written in Julia as

SqrSequence(n) = map(x->x*x, 1:n)

though this has the disadvantage that map will greedily process the whole 1:n range and return an array, i.e. this potentially uses a lot of memory. Is there actually a lazy version of map in Julia? I couldn’t find one, also not in the package IterTools. I did find Base.Generator, but it’s not exported so I guess one shouldn’t use it. Also it seems to return Any for eltype. Any other options for a lazy map?

[leiteiro]

OK. Give me a lazy map and I am happy.

[traktofon]

After checking https://pkg.julialang.org/, I see that the package

https://github.com/MikeInnes/Lazy.jl

has a lazymap, but it appears that it just delays building the array – once you run through the iterator, you still pay the memory price. Also eltype is still Any:

julia> using Lazy

julia> SqrSequence(n) = lazymap(x->x*x, 1:n)
SqrSequence (generic function with 1 method)

julia> s = SqrSequence(6);

julia> eltype(s)  # :( 
Any

Personally I think it should be possible to make a more light-weight lazymap. If there is really none, I’ll probably write one.

[traktofon]

Here’s a minimal implementation:

module LazyMaps

export LazyMap

struct LazyMap{F,I,T}
   fn   :: F
   iter :: I
end

function LazyMap(fn::F, iter::I) where {F,I}
   argtype = eltype(iter)
   T = Core.Inference.return_type(fn, (argtype,))
   LazyMap{F,I,T}(fn, iter)
end

Base.eltype(::LazyMap{F,I,T}) where {F,I,T} = T
Base.length(m::LazyMap) = length(m.iter)
Base.start(m::LazyMap) = start(m.iter)
Base.done(m::LazyMap, state) = done(m.iter, state)
Base.next(m::LazyMap, state) = begin; x,newstate = next(m.iter,state); (m.fn(x),newstate); end

end # module

Example usage:

julia> using LazyMaps

julia> SqrSequence(n) = LazyMap(x->x*x, 1:n)
SqrSequence (generic function with 1 method)

julia> s = SqrSequence(6)
LazyMaps.LazyMap{##1#2,UnitRange{Int64},Int64}(#1, 1:6)

julia> eltype(s)
Int64

julia> for x in s; print(x, " "); end
1 4 9 16 25 36

But I’m sure a better version of this already exists somewhere…

[bramtayl]

A lazy map is simply Base.Generator

[stevengj]

@bramtayl

A lazy map is simply Base.Generator

i.e. you can construct a new (lazy) iterator via e.g. (x*x for x in 1:n)

[bramtayl]

or with Base.Generator(x -> x*x, 1:n)

[traktofon]

@stevengj

(x*x for x in 1:n)

D’oh! Thanks.