# Concatenate Generators

**URL:** <https://discourse.julialang.org/t/concatenate-generators/59653>\
**Category:** Performance\
**Created:** [April 20, 2021, 9:57am UTC](https://discourse.julialang.org/t/concatenate-generators/59653 "2021-04-20T09:57:17Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![etienne\_dg](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@etienne\_dg](https://discourse.julialang.org/u/etienne_dg)\
**Post date:** [April 20, 2021, 9:57am UTC](https://discourse.julialang.org/t/concatenate-generators/59653/1 "2021-04-20T09:57:17Z")

</div>

Hi,  
I try to initialize a vector in two parts, for example, a vector with fifty 0’s then fifty 1’s.  
I can create it in a loop, but it is not very concise:

```julia
function foo1()
    a = Array{Int64}(undef, 100)
    for i in 1:50
        a[i]=0
    end
    for i in 51:100
        a[i]=1
    end
    return a
end

julia> @btime foo1()
  189.182 ns (1 allocation: 896 bytes)

```

I also can use vcat, but then, unnecessary allocations happen.

```julia
julia> @btime [zeros(Int, 50); ones(Int, 50)]
  341.602 ns (3 allocations: 1.84 KiB)

```

However, when working with ranges, vcat is smart enough to not allocate in the vcat:

```julia
julia> @btime [1:50;1:50]
  155.489 ns (1 allocation: 896 bytes)

```

I tried to vcat generators like this, but it did not work:

```julia
vcat((0 for i in 1:50), (1 for i in 1:50))

```

I also tried this, but it is surprisingly allocating a lot (and it is pretty ugly for such a simple task):

```julia
julia> @btime collect(Iterators.Flatten(((0 for i in 1:50), (1 for i in 1:50))))
  16.240 μs (313 allocations: 16.50 KiB)

```

Is there a way to concatenate generators properly and without allocating ?

```julia
julia> VERSION
v"1.6.0"

```

---

<div class="post-metadata">

**Author:** ![HenrikM](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrikm/32/17975_2.png) [@HenrikM](https://discourse.julialang.org/u/HenrikM)\
**Post date:** [April 20, 2021, 10:29am UTC](https://discourse.julialang.org/t/concatenate-generators/59653/2 "2021-04-20T10:29:29Z")

</div>

How about

```julia
[x>50 ? 0 : 1 for x=1:100]

```

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 20, 2021, 10:48am UTC](https://discourse.julialang.org/t/concatenate-generators/59653/3 "2021-04-20T10:48:07Z")

</div>

```julia
function foo(N)
    a = Vector{Int64}(undef, 2N)
    a[1:N] .= 0
    a[N+1:end] .= 1
    return a
end

# or, a little bit slower

function bar(N)
    a = zeros(Int, 2N)
    a[N+1:end] .= 1
    return a
end

```

---

<div class="post-metadata">

**Author:** ![MarcMush](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcmush/32/18006_2.png) [@MarcMush](https://discourse.julialang.org/u/MarcMush)\
**Post date:** [April 20, 2021, 11:15am UTC](https://discourse.julialang.org/t/concatenate-generators/59653/4 "2021-04-20T11:15:23Z")

</div>

you can have zero-step ranges with `StepRangeLen(start, step, length)`

```julia
julia> @btime [StepRangeLen(0,0,50); StepRangeLen(1,0,50)];
  65.886 ns (1 allocation: 896 bytes)

julia> @btime [1:50;1:50];
  64.460 ns (1 allocation: 896 bytes)

```

---

<div class="post-metadata">

**Author:** ![etienne\_dg](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@etienne\_dg](https://discourse.julialang.org/u/etienne_dg)\
**Post date:** [April 20, 2021, 12:21pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/5 "2021-04-20T12:21:26Z")

</div>

Ok, very interesting solutions so far !

> [@HenrikM](#):
>
> `[x>50 ? 0 : 1 for x=1:100]`

Works very well for this example, nice solution.

> [@DNF](#):
>
> ```julia
> function foo(N)
> a = Vector{Int64}(undef, 2N)
> a[1:N] .= 0
> a[N+1:end] .= 1
> return a
> end
> 
> ```

I thought that slicing would make copy, but I guess this is not the case here because it is a lvalue. I think this is the cleaner solution.

> [@MarcMush](#):
>
> you can have zero-step ranges with `StepRangeLen(start, step, length)`

I knew about the arguments length, step ant stop of range, but I didn’t know about this function, nice !

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 20, 2021, 1:01pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/7 "2021-04-20T13:01:59Z")

</div>

> [@etienne\_dg](#):
>
> I thought that slicing would make copy

Slicing syntax on the left hand side of an assignment does not create a copy, but calls the `setindex!` function, and mutates a view of the array the supplied indices.

> [@rafael.guerra](#):
>
> `@btime zeros(Int64,100)[51:100] .= 1`

This returns a view of the last 50 indices, not the entire vector.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [April 20, 2021, 1:04pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/8 "2021-04-20T13:04:07Z")

</div>

Oops, you are right… deleting it 🙂

---

<div class="post-metadata">

**Author:** ![etienne\_dg](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@etienne\_dg](https://discourse.julialang.org/u/etienne_dg)\
**Post date:** [April 20, 2021, 1:08pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/9 "2021-04-20T13:08:32Z")

</div>

> [@DNF](#):
>
> Slicing syntax on the left hand side of an assignment does not create a copy, but calls the `setindex!` function, and mutates a view of the array the supplied indices.

Yes, this makes sense, thank you very much !

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [April 20, 2021, 1:14pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/10 "2021-04-20T13:14:21Z")

</div>

> [@etienne\_dg](#):
>
> I also tried this, but it is surprisingly allocating a lot (and it is pretty ugly for such a simple task):
> 
> ```julia
> julia> @btime collect(Iterators.Flatten(((0 for i in 1:50), (1 for i in 1:50))))
> 16.240 μs (313 allocations: 16.50 KiB)
> 
> ```

This bit did not need the `collect`, unless you really need a `Vector` instead of a generator. However, it seems that even without the `collect` it allocates a lot, what is really surprising to me.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [April 20, 2021, 1:19pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/11 "2021-04-20T13:19:48Z")

</div>

The following creates a bit vector of 0,1. Not the fastest but simple:

```julia
julia> @btime (1:100) .> 50
  192.980 ns (2 allocations: 128 bytes)

```

---

<div class="post-metadata">

**Author:** ![etienne\_dg](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@etienne\_dg](https://discourse.julialang.org/u/etienne_dg)\
**Post date:** [April 20, 2021, 1:19pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/12 "2021-04-20T13:19:56Z")

</div>

I really need a Vector, but personally, I have no allocations without the `collect`.

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [April 20, 2021, 1:24pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/13 "2021-04-20T13:24:53Z")

</div>

ah, I am using Julia 1.5.3, you probably are using 1.6?

Well, if you need a `Vector` then you will always need at least one allocation of all its elements.

---

<div class="post-metadata">

**Author:** ![mschauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mschauer/32/13946_2.png) [@mschauer](https://discourse.julialang.org/u/mschauer)\
**Post date:** [April 20, 2021, 1:33pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/14 "2021-04-20T13:33:08Z")

</div>

One issue is that in `Iterators.Flatten(((0 for i in 1:50), (1 for i in 1:50))))` the `eltype`-trait mechanism has given up and returns `Any`

```julia
julia> eltype(Iterators.Flatten(((0 for i in 1:50), (1 for i in 1:50))))
Any

```

that doesn’t matter for iteration, but for collect.

---

<div class="post-metadata">

**Author:** ![etienne\_dg](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@etienne\_dg](https://discourse.julialang.org/u/etienne_dg)\
**Post date:** [April 20, 2021, 1:34pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/15 "2021-04-20T13:34:54Z")

</div>

Yes I’m using 1.6. I have no problem with the allocations proposed in the previous examples, as they do not allocate more than needed. I still don’t understand where the allocations in this specific example comes from.

Edit :

> [@mschauer](#):
>
> One issue is that in `Iterators.Flatten(((0 for i in 1:50), (1 for i in 1:50))))` the `eltype` -trait mechanism has given up and returns `Any`

Oh, I see, maybe there is a place for improvement here ?

---

<div class="post-metadata">

**Author:** ![mschauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mschauer/32/13946_2.png) [@mschauer](https://discourse.julialang.org/u/mschauer)\
**Post date:** [April 20, 2021, 1:39pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/16 "2021-04-20T13:39:20Z")

</div>

Not so much, generators save anonymous functions for the inner expression (here a function giving 0) and one runs into the issue of return type inference of functions in Julia

```julia
julia> G = (0 for i in 1:50)
Base.Generator{UnitRange{Int64}, var"#13#14"}(var"#13#14"(), 1:50)

julia> G.f
#13 (generic function with 1 method)

julia> G.f(1)
0

```

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [April 20, 2021, 1:58pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/17 "2021-04-20T13:58:31Z")

</div>

This one is cool:

```julia
@btime round.(LinRange(0,1,100))
  69.485 ns (1 allocation: 896 bytes)

```

and better:

```julia
@btime (sign.(-49:50) .+ 1).÷2
  57.986 ns (1 allocation: 896 bytes)

```

---

<div class="post-metadata">

**Author:** ![etienne\_dg](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@etienne\_dg](https://discourse.julialang.org/u/etienne_dg)\
**Post date:** [April 20, 2021, 2:36pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/18 "2021-04-20T14:36:41Z")

</div>

I don’t see where the problem is, julia seems to be able to infer the return type ?

```julia
julia> Base.return_types(G.f, (eltype(G.iter),))
1-element Vector{Any}:
 Int64

```

---

<div class="post-metadata">

**Author:** ![simeonschaub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simeonschaub/32/216566_2.png) [@simeonschaub](https://discourse.julialang.org/u/simeonschaub)\
**Post date:** [April 20, 2021, 2:54pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/19 "2021-04-20T14:54:58Z")

</div>

As a workaround, you could use e.g.

```julia
Iterators.flatten(map(x -> (x for i in 1:50), (0, 1)))

```

---

<div class="post-metadata">

**Author:** ![etienne\_dg](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@etienne\_dg](https://discourse.julialang.org/u/etienne_dg)\
**Post date:** [April 20, 2021, 3:01pm UTC](https://discourse.julialang.org/t/concatenate-generators/59653/20 "2021-04-20T15:01:00Z")

</div>

For me, it falls in the same issue, the type inferred is `Any`.