Creating Generators


I’ve been playing around with Julia lately and have been enjoying it. I have come from python and as a learning exercise was looking at re-writing some code I had written to do least significant bit embedding.

I’m wondering if my re-write below is the best way to write a more complicated generator or if there are better or more efficient ways to implement one.

Apologies if I’ve just missed something obvious.

Original python code below:

def bit_generator(s):
    for x in s:
        a = ord(x)

        i = 0
        while i < 7:
            yield a & 1
            a = a >> 1  # bit shifting embeds character backwards
            i += 1

    # signify end with 14 zeros (double ascii null)
    for x in xrange(14):
           yield 0

Julia equivalent (without the nulls at the end). This is where I missed yield as I ended up creating two generators to achieve the same - I could have easily just missed something here, but this is what I came up with.

type OrdGenerator

Base.start(x::OrdGenerator) = 1
Base.done(x::OrdGenerator, state) = length(x.s) == state - 1, state) = Int8(x.s[state]), state + 1

type BitGenerator

Base.start(x::BitGenerator) = 1
Base.done(x::BitGenerator, state) = state > 7

function, state)
    a = x.b & 1
    x.b = x.b >> 1
    a, state + 1

s = "word? w"
println("$s : length = $(length(s))")
for (i, x) in enumerate(OrdGenerator(s))
    println("$i: $x | $(bin(x))")
    for (j, bit) in enumerate(BitGenerator(x))
        println("  $j: $bit")

Any comments or advice would be appreciated.

From generator (with Channel) to iterator

The best way to build a “complicated generator” is to make a type which implements the iterator interface. That’s explained very well in the manual:


Thanks Chris.

I think my attempt above does use the iterator interface - happy that I’ve used the best method to create one.

After seeing your response and thinking about this a bit more, my main issue was probably more around the fact that I had to create two generators to achieve what I wanted as there was a nested while under the for.

I ended up working out that you can create a type for the state variable that has an accumulator, which works (see code below), but I’m still wondering if this approach is reasonable or what other people do?

Also, by “more complicated generator”, I really meant anything that couldn’t be easily achieved in something like a list comprehension.

type Generator

type State

Base.start(x::Generator) = State(1, 0, 0)
Base.done(x::Generator, state::State) = (length(x.s), 7) == (state.i, state.index_bit)

function, state::State)
    if state.index_bit == 0
        state = State(state.i, state.index_bit + 1, Int8(x.s[state.i]))
    elseif state.index_bit == 7
        state = State(state.i + 1, 1, Int8(x.s[state.i + 1]))
        state = State(state.i, state.index_bit + 1, state.acc >> 1)
    println("$(state.acc & 1) | $state")
    state.acc & 1, state
s = "here"
for (i, x) in enumerate(Generator(s)) end


Have you seen Channels? They are the direct Julia equivalent of Python generators.

#5 Thanks for the response.

I had looked at Tasks but as they were used in parallel computing, I thought they might be overkill for what I was doing.

I tried to create a generator using Channels (running Julia v0.5.2) this morning with no luck. I had success with the Tasks equivalent:

bit_generator(s::String) = Task() do
    for x in s
        a = Int8(x[1])
        i = 0
        while i < 7
            produce(a & 1)
            a = a >> 1
            i += 1
    for x in 1:14

for bit in bit_generator("text") 
    @printf("%d", bit)

My Channels attempt was essentially replacing as per below:

Tasks() do -> Channels() do c
produce(y) -> push!(c, y)

but I get the following error.

MethodError: Cannot `convert` an object of type ##45#46{String} to an object of type Channel{T}
This may have arisen from a call to the constructor Channel{T}(...),
since type constructors fall back to convert methods.

 in anonymous at ./<missing>:?

What am I missing?


Task is the right way to do this on 0.5; it is only on 0.6 that the concepts of Channel and Task have been merged.

#7 Thanks. I suspected that might be the case.



I also looking for a way to define a generator using Channel

Generator1() = Channel() do c
    i = 1
        push!(c, i)
        i = i + 1

function Generator2()
    c = Channel{Int}(1)
    i = 1
        push!(c, i)
        i = i + 1

# println(Generator1())

for i in Generator1()
    @show i

My issue is that I don’t like Generator1 to be a Channel of Any I’d prefer it to return a Channel of Int.

I tried to implement such an idea in Generator2 but nothing is displayed on console.

Any idea?

Generator1() = Channel(ctype=Int) do c
           i = 1
               push!(c, i)
               i = i + 1

The above code is calling this constructor Channel(func::Function; ctype=Any, csize=0, taskref=nothing). The first argument comes from the do syntax and the rest of the arguments can be used normally.