# Macro for Iterator

**URL:** <https://discourse.julialang.org/t/macro-for-iterator/5090>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [July 27, 2017, 11:18am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090 "2017-07-27T11:18:28Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![leiteiro](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@leiteiro](https://discourse.julialang.org/u/leiteiro)\
**Post date:** [July 27, 2017, 11:18am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/1 "2017-07-27T11:18:28Z")

</div>

Given a mutable struct H and a function G(S::H) I want to  
create this iterator:

```
Base.eltype(::Type{H}) = Int
Base.length(S::H) = S.a
Base.start(::H) = 1
Base.done(S::H, state) = state > S.a
Base.next(S::H, state) = (G(S), state + 1)

```

No problem so far. However I have to repeat this for many different  
pairs (H, G) over and over again.

Since the docs say macros allow the programmer to generate and  
include fragments of customized code I think they could help here.

How would such a macro look like?

For example: if H is the struct HelloState and the function G the  
HelloGenerator below …

```
module HelloModule

mutable struct HelloState
    a::Int
    b::Int
end

function HelloGenerator(S::HelloState)
    S.b += 1; "Hello" * ^("!", S.b) * " "
end

```

… this macro would expand to the next 5 lines below:

```
Base.eltype(::Type{HelloState}) = Int
Base.length(S::HelloState) = S.a
Base.start(::HelloState) = 1
Base.done(S::HelloState, state) = state > S.a
Base.next(S::HelloState, state) = (HelloGenerator(S), state + 1)

Hello(count) = HelloState(count, 0)
for i in Hello(6) print(i) end
end # module

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 27, 2017, 11:42am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/2 "2017-07-27T11:42:27Z")

</div>

I don’t think you need macros (unless I am missing something about your problem). Just make all types a subtype for some abstract type, and define the methods above for that type. Then it will work for all concrete subtypes.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 27, 2017, 7:33pm UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/3 "2017-07-27T19:33:45Z")

</div>

> [@leiteiro](#):
>
> `Base.next(S::H, state) = (G(S), state + 1)`

It seems a bit odd to define a different iterator for each function `G`. Are you sure you don’t just want an iterator that returns `S` and then use it with e.g. `map` or `mapreduce` to combine it with arbitrary functions `G`?

(Usually, `next` should not mutate the iterator object, only the `state`, though there are a few exceptions like `eachline` where the iterator has a side effect on the iterator object. It’s better to only modify the `state`, e.g. so you can have multiple/nested loops over the same iterator object.)

Iterator and generator concepts in Julia work a bit differently than they do in e.g. Python if you are coming from there.

---

<div class="post-metadata">

**Author:** ![leiteiro](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@leiteiro](https://discourse.julialang.org/u/leiteiro)\
**Post date:** [July 28, 2017, 7:27am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/4 "2017-07-28T07:27:37Z")

</div>

Interesting idea. Let’s see if I have understood.  
First I have to change the specification to:

```
Base.next(S::H, state) = (S.G(S), state + 1)

```

With a different example I than get:

```
abstract type Sequence end

Base.eltype(::Type{Sequence}) = Int
Base.length(S::Sequence) = S.count
Base.start(::Sequence) = 1
Base.done(S::Sequence, state) = state > S.count
Base.next(S::Sequence, state) = (S.f(S), state + 1)

mutable struct SqrState <: Sequence
	count::Int
	x::Int
	f::Function
end

function SqrGenerator(S::SqrState)
	S.x += 1; S.x * S.x
end

SqrSequence(count) = SqrState(count, 0, SqrGenerator)
for s in SqrSequence(6) print(s, " ") end

```

Suggestions for improvement?

---

<div class="post-metadata">

**Author:** ![leiteiro](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@leiteiro](https://discourse.julialang.org/u/leiteiro)\
**Post date:** [July 28, 2017, 7:28am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/5 "2017-07-28T07:28:31Z")

</div>

I want to ‘hide’ the iterator and not to make it  
explicite like it has to be with mapreduce.

I think of G as a ‘big’ state machine with states  
in H which is only triggered by the small state  
machine ‘iterator’.

It’s clear to me that I do not yet know how to  
express this idea in a ‘Julian way’. Seems to be  
much simpler to do in an object oriented setup.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 29, 2017, 2:43am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/6 "2017-07-29T02:43:10Z")

</div>

> [@leiteiro](#):
>
> I want to ‘hide’ the iterator and not to make it  
> explicite like it has to be with mapreduce.

Can you give a concrete example of what you would like to do?

---

<div class="post-metadata">

**Author:** ![leiteiro](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@leiteiro](https://discourse.julialang.org/u/leiteiro)\
**Post date:** [July 29, 2017, 6:49am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/7 "2017-07-29T06:49:00Z")

</div>

“I want to ‘hide’ the iterator and not to make it  
explicit like it has to be with mapreduce.”

It is intended to be used in a user API. The user needs  
not to see the iteration details.

In fact the proposal of Tamas Papp does accomplish this.  
See my example based on it: In the constructor

```
SqrSequence(count) = SqrState(count, 0, SqrGenerator) 

```

there is no explicit mention of an iterator.

I will mark Tamas proposal as a solution. What’s not nice  
is that the constructor has to set the initial values explicitly,  
i.e. that there is no way to give the struct default values  
which would make it more elegant. This has been discussed  
[here](https://github.com/JuliaLang/julia/issues/10146) but there seems to be no solution yet.

A different solution could be based on the produce/consume  
pattern which has unfortunately been destroyed in Julia 0.6,  
see this [discussion](https://github.com/JuliaLang/julia/issues/20927).

---

<div class="post-metadata">

**Author:** ![traktofon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/traktofon/32/591_2.png) [@traktofon](https://discourse.julialang.org/u/traktofon)\
**Post date:** [July 29, 2017, 7:19am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/8 "2017-07-29T07:19:37Z")

</div>

Hi @leiteiro,  
note that in Julia it is not necessary (and arguably counterproductive) to put function fields into your objects. Your previous code proposal can be rewritten in a more “Julian” way as follows (I think this is what @Tamas_Papp had in mind):

```julia
abstract type Sequence end

Base.eltype(::Type{Sequence}) = Int
Base.length(S::Sequence) = S.count
Base.start(::Sequence) = 1
Base.done(S::Sequence, state) = state > S.count
Base.next(S::Sequence, state) = (G(S), state + 1)

mutable struct SqrState <: Sequence
   count::Int
   x::Int
end

function G(S::SqrState)
   S.x += 1; S.x * S.x
end

SqrSequence(count) = SqrState(count, 0)
for s in SqrSequence(6) print(s, " ") end

```

The function `G`, which is called in your `next` method, is specialized on the _concrete_ iterator state type. To add another iterator, you then need to define another concrete state type, and add another method for `G`, as in the following example:

```julia
mutable struct ExpState <: Sequence
   count::Int
   x::Int
end

function G(S::ExpState)
   S.x += 1; exp(S.x)
end

ExpSequence(count) = ExpState(count, 0)
for s in ExpSequence(6) print(s, " ") end

```

---

<div class="post-metadata">

**Author:** ![traktofon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/traktofon/32/591_2.png) [@traktofon](https://discourse.julialang.org/u/traktofon)\
**Post date:** [July 29, 2017, 7:41am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/9 "2017-07-29T07:41:52Z")

</div>

Going slightly off topic, the kind of generator/iterators discussed here can quite simply be written in Julia as

```julia
SqrSequence(n) = map(x->x*x, 1:n)

```

though this has the disadvantage that `map` will greedily process the whole `1:n` range and return an array, i.e. this potentially uses a lot of memory. Is there actually a _lazy_ version of `map` in Julia? I couldn’t find one, also not in the package `IterTools`. I did find `Base.Generator`, but it’s not exported so I guess one shouldn’t use it. Also it seems to return `Any` for `eltype`. Any other options for a lazy `map`?

---

<div class="post-metadata">

**Author:** ![leiteiro](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@leiteiro](https://discourse.julialang.org/u/leiteiro)\
**Post date:** [July 29, 2017, 7:47am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/10 "2017-07-29T07:47:14Z")

</div>

OK. Give me a lazy map and I am happy.

---

<div class="post-metadata">

**Author:** ![traktofon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/traktofon/32/591_2.png) [@traktofon](https://discourse.julialang.org/u/traktofon)\
**Post date:** [July 29, 2017, 8:04am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/11 "2017-07-29T08:04:57Z")

</div>

After checking [https://pkg.julialang.org/](https://pkg.julialang.org/), I see that the package

[https://github.com/MikeInnes/Lazy.jl](https://github.com/MikeInnes/Lazy.jl)

has a `lazymap`, but it appears that it just delays building the array – once you run through the iterator, you still pay the memory price. Also `eltype` is still `Any`:

```julia
julia> using Lazy

julia> SqrSequence(n) = lazymap(x->x*x, 1:n)
SqrSequence (generic function with 1 method)

julia> s = SqrSequence(6);

julia> eltype(s) # :( 
Any

```

Personally I think it should be possible to make a more light-weight `lazymap`. If there is really none, I’ll probably write one.

---

<div class="post-metadata">

**Author:** ![traktofon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/traktofon/32/591_2.png) [@traktofon](https://discourse.julialang.org/u/traktofon)\
**Post date:** [July 29, 2017, 8:38am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/12 "2017-07-29T08:38:20Z")

</div>

Here’s a minimal implementation:

```julia
module LazyMaps

export LazyMap

struct LazyMap{F,I,T}
   fn :: F
   iter :: I
end

function LazyMap(fn::F, iter::I) where {F,I}
   argtype = eltype(iter)
   T = Core.Inference.return_type(fn, (argtype,))
   LazyMap{F,I,T}(fn, iter)
end

Base.eltype(::LazyMap{F,I,T}) where {F,I,T} = T
Base.length(m::LazyMap) = length(m.iter)
Base.start(m::LazyMap) = start(m.iter)
Base.done(m::LazyMap, state) = done(m.iter, state)
Base.next(m::LazyMap, state) = begin; x,newstate = next(m.iter,state); (m.fn(x),newstate); end

end # module

```

Example usage:

```julia
julia> using LazyMaps

julia> SqrSequence(n) = LazyMap(x->x*x, 1:n)
SqrSequence (generic function with 1 method)

julia> s = SqrSequence(6)
LazyMaps.LazyMap{##1#2,UnitRange{Int64},Int64}(#1, 1:6)

julia> eltype(s)
Int64

julia> for x in s; print(x, " "); end
1 4 9 16 25 36

```

But I’m sure a better version of this already exists somewhere…

---

<div class="post-metadata">

**Author:** ![bramtayl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bramtayl/32/3614_2.png) [@bramtayl](https://discourse.julialang.org/u/bramtayl)\
**Post date:** [July 29, 2017, 11:38am UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/13 "2017-07-29T11:38:44Z")

</div>

A lazy map is simply `Base.Generator`

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 29, 2017, 12:44pm UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/14 "2017-07-29T12:44:59Z")

</div>

> [@bramtayl](#):
>
> A lazy map is simply Base.Generator

i.e. you can construct a new (lazy) iterator via e.g. `(x*x for x in 1:n)`

---

<div class="post-metadata">

**Author:** ![bramtayl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bramtayl/32/3614_2.png) [@bramtayl](https://discourse.julialang.org/u/bramtayl)\
**Post date:** [July 29, 2017, 1:46pm UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/15 "2017-07-29T13:46:27Z")

</div>

or with `Base.Generator(x -> x*x, 1:n)`

---

<div class="post-metadata">

**Author:** ![traktofon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/traktofon/32/591_2.png) [@traktofon](https://discourse.julialang.org/u/traktofon)\
**Post date:** [July 29, 2017, 4:32pm UTC](https://discourse.julialang.org/t/macro-for-iterator/5090/16 "2017-07-29T16:32:40Z")

</div>

> [@stevengj](#):
>
> (x\*x for x in 1:n)

D’oh! Thanks.