# Generator as an "Array Initializer?"

**URL:** <https://discourse.julialang.org/t/generator-as-an-array-initializer/93379>\
**Category:** Internals & Design\
**Tags:** question\
**Created:** [January 23, 2023, 6:14am UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379 "2023-01-23T06:14:35Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [January 23, 2023, 6:14am UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/1 "2023-01-23T06:14:35Z")

</div>

(inspired by [this discussion](https://discourse.julialang.org/t/what-about-an-undefs-function-in-base/92690/59))

Would it make sense for the `Array` constructor to accept a generator as an “array initializer?”

```julia
Array{Float64}(i+j for i=1:m, j=1:n)

```

The downside is that it’s redundant with the `:typed_comprehension` expression, e.g. `Float64[...]`, but the upside is that it’s consistent with other constructors such as `Tuple`, and it provides an idiom for non-standard arrays to use generators (while avoiding the `undef` initialization with `for` loop idiom), such as:

```julia
using CUDA
CuArray{Float64}(i+j for i=1:m, j=1:n)

```

or even:

```julia
using OffsetArrays
ℱ = OffsetArray(exp(-2π*im*m*n/N) for m=0:N-1, n=0:N-1)

```

and I would presume `OffsetArray` would set up the indexing as needed by accessing `generator.iter.iterators`.

Presently some packages offer helper macros instead, such as:

```julia
using StaticArrays
@SArray[x for x=1:5]

```

but this feels rather unidiomatic, and is visually something of an [oddity](https://discourse.julialang.org/t/what-about-an-undefs-function-in-base/92690/62).

Thoughts?

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [January 23, 2023, 11:56am UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/2 "2023-01-23T11:56:45Z")

</div>

I agree! When I first learned about the Array constructor, I found it very confusing that it couldn’t initialize the array with arbitrary values. Accepting a generator would be nice.

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [January 23, 2023, 2:17pm UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/3 "2023-01-23T14:17:37Z")

</div>

This sounds like great idea.

It is really just a special case of `T(x)` turning `x` into a `T`, so the fact that it is not already implemented makes me think that it was probably discussed long ago, and there was some good reason not to do it. (But I cannot think of any such reason myself.)

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [January 24, 2023, 4:55pm UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/4 "2023-01-24T16:55:29Z")

</div>

This is already close to the `collect` method.

```julia
julia> m = 10; n = 5
5

julia> collect(Float64, i+j for i in 1:m, j in 1:n)
10×5 Matrix{Float64}:
  2.0 3.0 4.0 5.0 6.0
  3.0 4.0 5.0 6.0 7.0
  4.0 5.0 6.0 7.0 8.0
  5.0 6.0 7.0 8.0 9.0
  6.0 7.0 8.0 9.0 10.0
  7.0 8.0 9.0 10.0 11.0
  8.0 9.0 10.0 11.0 12.0
  9.0 10.0 11.0 12.0 13.0
 10.0 11.0 12.0 13.0 14.0
 11.0 12.0 13.0 14.0 15.0

```

To avoid type piracy, we might try the following wrapper generator approach.

```julia
julia> struct GeneratorInit{T <: Base.Generator}
           gen::T
       end

julia> geninit(gen) = GeneratorInit(gen)
geninit (generic function with 1 method)

julia> function (::Type{A})(init::GeneratorInit) where {T, A <: AbstractArray{T}}
           A(collect(T, init.gen))
       end

julia> Array{Float64}(geninit(i+j for i in 1:5, j in 1:6))
5×6 Matrix{Float64}:
 2.0 3.0 4.0 5.0 6.0 7.0
 3.0 4.0 5.0 6.0 7.0 8.0
 4.0 5.0 6.0 7.0 8.0 9.0
 5.0 6.0 7.0 8.0 9.0 10.0
 6.0 7.0 8.0 9.0 10.0 11.0

```

Perhaps a function that accepts a `CartesianIndex` may be useful as well?

```julia
julia> function Array(f::Function, dims...)
           ci = CartesianIndices(dims)
           map(f, ci)
       end
Array

julia> Array(5, 6) do ci
           ci[1] + ci[2]
       end
5×6 Matrix{Int64}:
 2 3 4 5 6 7
 3 4 5 6 7 8
 4 5 6 7 8 9
 5 6 7 8 9 10
 6 7 8 9 10 11

julia> function Array{T}(f::Function, dims...) where T
           ci = CartesianIndices(dims)
           map(ci) do i
              T(f(ci))
           end
       end

julia> Array{Float64}(5, 6) do ci
           ci[1] + ci[2]
       end
5×6 Matrix{Float64}:
 2.0 3.0 4.0 5.0 6.0 7.0
 3.0 4.0 5.0 6.0 7.0 8.0
 4.0 5.0 6.0 7.0 8.0 9.0
 5.0 6.0 7.0 8.0 9.0 10.0
 6.0 7.0 8.0 9.0 10.0 11.0

```

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [January 24, 2023, 8:54pm UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/5 "2023-01-24T20:54:40Z")

</div>

I added generator initializers to `ArrayInitializers#geninit` (geninit git branch of ArrayInitializers.jl).

```julia
julia> using Pkg

julia> pkg"add ArrayInitializers#geninit"
...
julia> using ArrayInitializers

julia> Array(geninit(x^2 for x in 2:5))
4-element Vector{Int64}:
  4
  9
 16
 25

julia> Array{Float64}(geninit(x^2 for x in 2:5))
4-element Vector{Float64}:
  4.0
  9.0
 16.0
 25.0

julia> Array{Float64}(geninit(x^2+y for x in 2:5, y in 1:10))
4×10 Matrix{Float64}:
  5.0 6.0 7.0 8.0 9.0 10.0 11.0 12.0 13.0 14.0
 10.0 11.0 12.0 13.0 14.0 15.0 16.0 17.0 18.0 19.0
 17.0 18.0 19.0 20.0 21.0 22.0 23.0 24.0 25.0 26.0
 26.0 27.0 28.0 29.0 30.0 31.0 32.0 33.0 34.0 35.0

julia> using OffsetArrays

julia> OffsetArray(geninit(x^2 for x in 1:2:9), 6:10)
5-element OffsetArray(::Vector{Int64}, 6:10) with eltype Int64 with indices 6:10:
  1
  9
 25
 49
 81

julia> OffsetArray{UInt8}(geninit(x^2 for x in 1:2:9), 6:10)
5-element OffsetArray(::Vector{UInt8}, 6:10) with eltype UInt8 with indices 6:10:
 0x01
 0x09
 0x19
 0x31
 0x51

```

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [January 25, 2023, 6:30am UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/6 "2023-01-25T06:30:51Z")

</div>

> [@Per](#):
>
> it was probably discussed long ago, and there was some good reason not to do it.

You’re right, this idea is too obvious; there has to be something wrong with it 😅

> [@mkitti](#):
>
> I added generator initializers

You move quick! My hesitation is that:

1. this feels more like it should be a language feature instead of being in a package, and
2. doesn’t this implementation allocate and fill an entire array before converting to the desired type, and therefore can be needlessly expensive?

On point (2), what inspired this idea to begin with was whether we could help avoid the `undef` allocation + `for` loop initialization idiom for non-standard arrays, but often why people might do that to begin with is because they want to avoid allocating unnecessary arrays. If we’re okay with unnecessary allocation, we can already call a constructor on a comprehension e.g. `SMatrix{m,n}([i+j for i=1:m, j=1:n])`

> [@mkitti](#):
>
> Perhaps a function that accepts a `CartesianIndex` may be useful as well?

Hm, that’s an interesting idea! why _shouldn’t_ user-defined functions be array initializers?

Although I’d probably prefer, instead of accepting a `CartesianIndices` object, to allow for a function signature that allows `Array(5, 6) do (i,j); i+j end`.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [January 25, 2023, 8:07am UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/7 "2023-01-25T08:07:34Z")

</div>

> [@uniment](#):
>
> You move quick! My hesitation is that:
> 
> 1. this feels more like it should be a language feature instead of being in a package, and
> 2. doesn’t this implementation allocate and fill an entire array before converting to the desired type, and therefore can be needlessly expensive?

1. Language features are really hard. This is really more of a standard API issue though, which is also very difficult to get right. As I explained before, it’s better to work on the design in a package first before trying to propose the pull request.
2. It does, but the composition of ArrayInitializers and StaticArrays has not been worked on. I think I might make my first “glue” package to cover this.

> [@uniment](#):
>
> Although I’d probably prefer, instead of accepting a `CartesianIndices` object, to allow for a function signature that allows `Array(5, 6) do (i,j); i+j end`.

I would provide the `CartesianIndex`. The `CartesianIndex` is just a wrapped tuple:

```julia
julia> CartesianIndex(2,3).I
(2, 3)

```

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [January 25, 2023, 2:05pm UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/8 "2023-01-25T14:05:16Z")

</div>

The fact that the element type is known in advance makes this initialiser much simpler than the `collect` function. In principle, all that is needed for it to work in Base is:

```julia
function (::Type{A})(itr::Base.Generator) where {A<:AbstractArray} 
    dest = A(undef, size(itr))
    copyto!(dest, itr)
end

```

One might want to add some error handling (telling the user to revert to `collect`) instead of just throwing a `MethodError` when the generator doesn’t have a `size`, but otherwise I really don’t think it has to be more complicated than this.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [January 25, 2023, 4:03pm UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/9 "2023-01-25T16:03:16Z")

</div>

We could use `applicable` and `hasmethod` to detect if `size` is applicable to the iterator within the generator.

```julia
julia> g = (x for x in 1:5)
Base.Generator{UnitRange{Int64}, typeof(identity)}(identity, 1:5)

julia> applicable(size, g.iter)
true

julia> gc = (x for x in Iterators.take(Iterators.cycle(1:5),9))
Base.Generator{Base.Iterators.Take{Base.Iterators.Cycle{UnitRange{Int64}}}, typeof(identity)}(identity, Base.Iterators.Take{Base.Iterators.Cycle{UnitRange{Int64}}}(Base.Iterators.Cycle{UnitRange{Int64}}(1:5), 9))

julia> applicable(size, gc.iter)
false

```

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [January 25, 2023, 4:41pm UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/10 "2023-01-25T16:41:48Z")

</div>

There’s also `Base.IteratorSize` which seems to be there for exactly this purpose. So one could do:

```julia
function (::Type{A})(itr::Base.Generator) where {A<:AbstractArray}
    dest = _array4itr(A, itr, Base.IteratorSize(itr))
    copyto!(dest, itr)
end

_array4itr(::Type{A}, itr, ::Base.HasShape) where {A} = A(undef, size(itr))
_array4itr(::Type{A}, itr, ::Base.HasLength) where {A} = A(undef, length(itr))
_array4itr(::Type{A}, itr, ::Base.SizeUnknown) where {A} = error("Cannot initialize an array for an iterator of unknown size.")

```

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [January 25, 2023, 10:20pm UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/11 "2023-01-25T22:20:34Z")

</div>

> [@mkitti](#):
>
> The `CartesianIndex` is just a wrapped tuple:

In that case, why not just use the tuple? At least the tuple is convenient to destructure. Do we get any dispatch benefits here?

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [January 25, 2023, 10:58pm UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/12 "2023-01-25T22:58:07Z")

</div>

With a `CartesianIndex` you can index directly into a multidimensional array. You cannot do that with a pure tuple.

```julia
julia> A = rand(5,3)
5×3 Matrix{Float64}:
 0.451517 0.152694 0.454544
 0.0387113 0.987212 0.294335
 0.369965 0.567338 0.797788
 0.661118 0.422781 0.601305
 0.984192 0.611997 0.794501

julia> I = CartesianIndex(5,3)
CartesianIndex(5, 3)

julia> A[(5,3)]
ERROR: ArgumentError: invalid index: (5, 3) of type Tuple{Int64, Int64}
...

julia> A[I]
0.794500894925653

```

Retrieving the tuple from the `CartesianIndex` is pretty easy:

```julia
julia> Tuple(I)
(5, 3)

julia> I.I
(5, 3)

julia> x, y = I.I
(5, 3)

julia> x
5

julia> y
3

```

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [January 26, 2023, 4:58am UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/13 "2023-01-26T04:58:59Z")

</div>

If I want to index directly into an existing multidimensional array though, wouldn’t I just do this?

```julia
Array(m, n) do (i, j); other_arr[i, j] end

```

If `CartesianIndex` was iterable this wouldn’t be a concern, but it’s not so I’d rather avoid it.

Then again, looking at [PR#23719](https://github.com/JuliaLang/julia/pull/23719)—which explains the rationale for preventing iteration as avoiding a performance trap—maybe it’s been fixed since then.

> **Benchmark showing CartesianIndex iteration looks ok now**
>
> ```julia
> julia> Base.iterate(itr::CartesianIndex, t=itr.I) = Iterators.peel(t)
> 
> julia> function mysum1(A)
> s = 0.0
> @inbounds for i in CartesianIndices(A)
> s += A[i]
> end
> s
> end
> mysum1 (generic function with 1 method)
> 
> julia> function mysum2(A)
> s = 0.0
> @inbounds for i in CartesianIndices(A)
> s += A[i...]
> end
> s
> end
> mysum2 (generic function with 1 method)
> 
> julia> A = rand(10,10,10,10,10,10);
> 
> # after warmup
> julia> @time mysum1(A)
> 0.001131 seconds (1 allocation: 16 bytes)
> 499454.20988074585
> 
> julia> @time mysum2(A)
> 0.001158 seconds (1 allocation: 16 bytes)
> 499454.20988074585
> 
> julia> @btime mysum1($A)
> 514.000 μs (0 allocations: 0 bytes)
> 499454.20988074585
> 
> julia> @btime mysum2($A)
> 514.000 μs (0 allocations: 0 bytes)
> 499454.20988074585
> 
> ```

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [January 26, 2023, 5:28am UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/14 "2023-01-26T05:28:41Z")

</div>

Oh, this looks like a good idea. [PR#48404](https://github.com/JuliaLang/julia/pull/48404)

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [January 26, 2023, 5:43am UTC](https://discourse.julialang.org/t/generator-as-an-array-initializer/93379/15 "2023-01-26T05:43:04Z")

</div>

> [@mkitti](#):
>
> `collect(Float64, i+j for i in 1:m, j in 1:n)`

This may be used to create an `OffsetArray` as well, by providing offset axes as the loop indices.

```julia
julia> using OffsetArrays

julia> collect(Float64, i+j for i in 1:3, j in Base.IdentityUnitRange(2:4))
3×3 OffsetArray(::Matrix{Float64}, 1:3, 2:4) with eltype Float64 with indices 1:3×2:4:
 3.0 4.0 5.0
 4.0 5.0 6.0
 5.0 6.0 7.0

```