# Macro for Iterator

Given a mutable struct H and a function G(S::H) I want to
create this iterator:

``````Base.eltype(::Type{H}) = Int
Base.length(S::H) = S.a
Base.start(::H) = 1
Base.done(S::H, state) = state > S.a
Base.next(S::H, state) = (G(S), state + 1)
``````

No problem so far. However I have to repeat this for many different
pairs (H, G) over and over again.

Since the docs say macros allow the programmer to generate and
include fragments of customized code I think they could help here.

How would such a macro look like?

For example: if H is the struct HelloState and the function G the
HelloGenerator below ā¦

``````module HelloModule

mutable struct HelloState
a::Int
b::Int
end

function HelloGenerator(S::HelloState)
S.b += 1; "Hello" * ^("!", S.b) * " "
end
``````

ā¦ this macro would expand to the next 5 lines below:

``````Base.eltype(::Type{HelloState}) = Int
Base.length(S::HelloState) = S.a
Base.start(::HelloState) = 1
Base.done(S::HelloState, state) = state > S.a
Base.next(S::HelloState, state) = (HelloGenerator(S), state + 1)

Hello(count) = HelloState(count, 0)
for i in Hello(6) print(i) end
end # module
``````

I donāt think you need macros (unless I am missing something about your problem). Just make all types a subtype for some abstract type, and define the methods above for that type. Then it will work for all concrete subtypes.

5 Likes

It seems a bit odd to define a different iterator for each function `G`. Are you sure you donāt just want an iterator that returns `S` and then use it with e.g. `map` or `mapreduce` to combine it with arbitrary functions `G`?

(Usually, `next` should not mutate the iterator object, only the `state`, though there are a few exceptions like `eachline` where the iterator has a side effect on the iterator object. Itās better to only modify the `state`, e.g. so you can have multiple/nested loops over the same iterator object.)

Iterator and generator concepts in Julia work a bit differently than they do in e.g. Python if you are coming from there.

1 Like

Interesting idea. Letās see if I have understood.
First I have to change the specification to:

``````Base.next(S::H, state) = (S.G(S), state + 1)
``````

With a different example I than get:

``````abstract type Sequence end

Base.eltype(::Type{Sequence}) = Int
Base.length(S::Sequence) = S.count
Base.start(::Sequence) = 1
Base.done(S::Sequence, state) = state > S.count
Base.next(S::Sequence, state) = (S.f(S), state + 1)

mutable struct SqrState <: Sequence
count::Int
x::Int
f::Function
end

function SqrGenerator(S::SqrState)
S.x += 1; S.x * S.x
end

SqrSequence(count) = SqrState(count, 0, SqrGenerator)
for s in SqrSequence(6) print(s, " ") end
``````

Suggestions for improvement?

I want to āhideā the iterator and not to make it
explicite like it has to be with mapreduce.

I think of G as a ābigā state machine with states
in H which is only triggered by the small state
machine āiteratorā.

Itās clear to me that I do not yet know how to
express this idea in a āJulian wayā. Seems to be
much simpler to do in an object oriented setup.

Can you give a concrete example of what you would like to do?

āI want to āhideā the iterator and not to make it
explicit like it has to be with mapreduce.ā

It is intended to be used in a user API. The user needs
not to see the iteration details.

In fact the proposal of Tamas Papp does accomplish this.
See my example based on it: In the constructor

``````SqrSequence(count) = SqrState(count, 0, SqrGenerator)
``````

there is no explicit mention of an iterator.

I will mark Tamas proposal as a solution. Whatās not nice
is that the constructor has to set the initial values explicitly,
i.e. that there is no way to give the struct default values
which would make it more elegant. This has been discussed
here but there seems to be no solution yet.

A different solution could be based on the produce/consume
pattern which has unfortunately been destroyed in Julia 0.6,
see this discussion.

Hi @leiteiro,
note that in Julia it is not necessary (and arguably counterproductive) to put function fields into your objects. Your previous code proposal can be rewritten in a more āJulianā way as follows (I think this is what @Tamas_Papp had in mind):

``````abstract type Sequence end

Base.eltype(::Type{Sequence}) = Int
Base.length(S::Sequence) = S.count
Base.start(::Sequence) = 1
Base.done(S::Sequence, state) = state > S.count
Base.next(S::Sequence, state) = (G(S), state + 1)

mutable struct SqrState <: Sequence
count::Int
x::Int
end

function G(S::SqrState)
S.x += 1; S.x * S.x
end

SqrSequence(count) = SqrState(count, 0)
for s in SqrSequence(6) print(s, " ") end
``````

The function `G`, which is called in your `next` method, is specialized on the concrete iterator state type. To add another iterator, you then need to define another concrete state type, and add another method for `G`, as in the following example:

``````mutable struct ExpState <: Sequence
count::Int
x::Int
end

function G(S::ExpState)
S.x += 1; exp(S.x)
end

ExpSequence(count) = ExpState(count, 0)
for s in ExpSequence(6) print(s, " ") end
``````

Going slightly off topic, the kind of generator/iterators discussed here can quite simply be written in Julia as

``````SqrSequence(n) = map(x->x*x, 1:n)
``````

though this has the disadvantage that `map` will greedily process the whole `1:n` range and return an array, i.e. this potentially uses a lot of memory. Is there actually a lazy version of `map` in Julia? I couldnāt find one, also not in the package `IterTools`. I did find `Base.Generator`, but itās not exported so I guess one shouldnāt use it. Also it seems to return `Any` for `eltype`. Any other options for a lazy `map`?

1 Like

OK. Give me a lazy map and I am happy.

After checking https://pkg.julialang.org/, I see that the package

https://github.com/MikeInnes/Lazy.jl

has a `lazymap`, but it appears that it just delays building the array ā once you run through the iterator, you still pay the memory price. Also `eltype` is still `Any`:

``````julia> using Lazy

julia> SqrSequence(n) = lazymap(x->x*x, 1:n)
SqrSequence (generic function with 1 method)

julia> s = SqrSequence(6);

julia> eltype(s)  # :(
Any
``````

Personally I think it should be possible to make a more light-weight `lazymap`. If there is really none, Iāll probably write one.

Hereās a minimal implementation:

``````module LazyMaps

export LazyMap

struct LazyMap{F,I,T}
fn   :: F
iter :: I
end

function LazyMap(fn::F, iter::I) where {F,I}
argtype = eltype(iter)
T = Core.Inference.return_type(fn, (argtype,))
LazyMap{F,I,T}(fn, iter)
end

Base.eltype(::LazyMap{F,I,T}) where {F,I,T} = T
Base.length(m::LazyMap) = length(m.iter)
Base.start(m::LazyMap) = start(m.iter)
Base.done(m::LazyMap, state) = done(m.iter, state)
Base.next(m::LazyMap, state) = begin; x,newstate = next(m.iter,state); (m.fn(x),newstate); end

end # module
``````

Example usage:

``````julia> using LazyMaps

julia> SqrSequence(n) = LazyMap(x->x*x, 1:n)
SqrSequence (generic function with 1 method)

julia> s = SqrSequence(6)
LazyMaps.LazyMap{##1#2,UnitRange{Int64},Int64}(#1, 1:6)

julia> eltype(s)
Int64

julia> for x in s; print(x, " "); end
1 4 9 16 25 36
``````

But Iām sure a better version of this already exists somewhereā¦

A lazy map is simply `Base.Generator`

i.e. you can construct a new (lazy) iterator via e.g. `(x*x for x in 1:n)`

1 Like

or with `Base.Generator(x -> x*x, 1:n)`

Dāoh! Thanks.