# \[ANN\] IndexFunArrays.jl - Fun with indices (and functions on them)

**URL:** <https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862>\
**Category:** Package Announcements\
**Tags:** package, array\
**Created:** [March 24, 2021, 11:54am UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862 "2021-03-24T11:54:03Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [March 24, 2021, 11:54am UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/1 "2021-03-24T11:54:03Z")

</div>

Hey,

[we](https://github.com/bionanoimaging) recently created [IndexFunArrays.jl](https://github.com/bionanoimaging/IndexFunArrays.jl).

You can try it with:

```julia
julia> ] add IndexFunArrays

```

Initially we wanted a tool calculating output arrays using the index position. For that purpose we created a `IndexFunArray` (with size independent constant memory allocation) which calculates the value once accessed with `getindex`. It needs an index function and a output size:

```julia
julia> z = IndexFunArray(x -> sum(abs2.(x)), (3, 3)) # directly using the constructor and supplying a function to store in the array
  3×3 IndexFunArray{Int64, 2, var"#184#185"}:
    2 5 10
    5 8 13
   10 13 18

julia> typeof(z) <: AbstractArray
true

julia> z[3,3] # use it like a normal array which then calculates the value
18

```

For convenience we already created several [distance related wrapper](https://bionanoimaging.github.io/IndexFunArrays.jl/dev/distance/) and some [window functions](https://bionanoimaging.github.io/IndexFunArrays.jl/dev/window/):

```julia
julia> rr2((4, 4))
4×4 IndexFunArray{Float64, 2, IndexFunArrays.var"#4#5"{Float64, Tuple{Float64, Float64}, Tuple{Int64, Int64}}}:
 8.0 5.0 4.0 5.0
 5.0 2.0 1.0 2.0
 4.0 1.0 0.0 1.0
 5.0 2.0 1.0 2.0

julia> rr2((3,3), offset=CtrCorner)
  3×3 IndexFunArray{Float64, 2, IndexFunArrays.var"#4#5"{Float64, Tuple{Float64, Float64}, Tuple{Int64, Int64}}}:
   0.0 1.0 4.0
   1.0 2.0 5.0
   4.0 5.0 8.0

julia> rr2((3,3), offset=(4,4))
3×3 IndexFunArray{Float64, 2, IndexFunArrays.var"#4#5"{Float64, Tuple{Int64, Int64}, Tuple{Int64, Int64}}}:
 18.0 13.0 10.0
 13.0 8.0 5.0
 10.0 5.0 2.0

```

The package works with other packages like `LazyArrays` or `CUDA`.  
We were curious if there isn’t something similar already? Maybe some of you know more about.

We would be happy about feedback!

Thanks,

Felix

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [March 24, 2021, 1:17pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/2 "2021-03-24T13:17:09Z")

</div>

Interesting!

> [@roflmaostc](#):
>
> We were curious if there isn’t something similar already?

If [#37648](https://github.com/JuliaLang/julia/pull/37648) gets merged you could do the following in plain Julia:

```julia
julia> z = (sum(abs2.((i,j))) for i in 1:3, j in 1:3);

julia> z[3,3]
18

```

or

```julia
julia> z = (sum(abs2.(Tuple(x))) for x in CartesianIndices((3, 3)));

julia> z[3,3]
18

```

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [March 24, 2021, 1:22pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/3 "2021-03-24T13:22:25Z")

</div>

That’s cool and would it have been present, we probably didn’t start this package 😅

But the important part for us are the convenient wrappers. We might remove the `IndexFunArray` and switch to the built-in one in the future.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 24, 2021, 1:34pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/4 "2021-03-24T13:34:04Z")

</div>

Thanks for sharing.  
Wondering if using the `map` function and `CartesianIndices` one couldn’t do some of this nice stuff?

For instance, your first example could be done as:

```julia
ix(n,m) = map(i-> [i[1],i[2]], CartesianIndices((1:n,1:m)))

map( x -> sum(abs2.(x)), ix(3,3) )

3×3 Matrix{Int64}:
  2 5 10
  5 8 13
 10 13 18

```

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [March 24, 2021, 1:37pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/5 "2021-03-24T13:37:50Z")

</div>

Yes, but mapping allocates memory.  
One then could use [MappedArrays.jl](https://github.com/JuliaArrays/MappedArrays.jl) but which does need an beforehand allocated array as input (which `IndexFunArray` does not need).

---

<div class="post-metadata">

**Author:** ![logankilpatrick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/logankilpatrick/32/38123_2.png) [@logankilpatrick](https://discourse.julialang.org/u/logankilpatrick)\
**Post date:** [March 24, 2021, 9:04pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/6 "2021-03-24T21:04:28Z")

</div>



---

<div class="post-metadata">

**Author:** ![logankilpatrick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/logankilpatrick/32/38123_2.png) [@logankilpatrick](https://discourse.julialang.org/u/logankilpatrick)\
**Post date:** [March 24, 2021, 11:19pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/7 "2021-03-24T23:19:53Z")

</div>



---

<div class="post-metadata">

**Author:** ![logankilpatrick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/logankilpatrick/32/38123_2.png) [@logankilpatrick](https://discourse.julialang.org/u/logankilpatrick)\
**Post date:** [March 24, 2021, 11:20pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/8 "2021-03-24T23:20:37Z")

</div>

Accidentally converted this to a PM on mobile, sorry about that!

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [March 25, 2021, 6:23am UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/9 "2021-03-25T06:23:33Z")

</div>

Cool package. I was reading over the src code and have a quick question: The definition

```julia
struct IndexFunArray{T, N, F} <: AbstractArray{T, N} where {F}

```

is parametrized by types `T, N, F` so shouldn’t it be `where {T, N, F} ` at the end of the line?

Another question I had was that the some functions have a signature where the argument is `IndexFunArray{T,N}`, e.g.

```julia
Base.getindex(A::IndexFunArray{T,N}, I::Vararg{Int, N}) where {T,N} = return A.generator(I)

```

but isn’t the object actually `IndexFunArray{T,N,F}` so how come multiple dispatch gets it correct?

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [March 25, 2021, 8:42am UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/10 "2021-03-25T08:42:49Z")

</div>

That’s I good question I don’t know either.

But we can try the following:

```julia
julia> struct Foo{T, N, F} <: AbstractArray{T, N} where {T, N, F}
           bar::Int
       end
ERROR: invalid subtyping in definition of Foo
Stacktrace:
 [1] top-level scope
   @ REPL[1]:1

julia> struct Foo{T, N, F} <: AbstractArray{T, N} where {F}
           bar::Int
       end

```

So the first one is not possible. My interpretation would be that the `T,N` are already _fixed_ by the `AbtractArray{T, N}` and therefore are not free anymore.  
In the documentation I could find this part about [Free variables](https://docs.julialang.org/en/v1/devdocs/types/#Free-variables).

And regarding the second one:

```julia
julia> AbstractArray{Int}
AbstractArray{Int64, N} where N

julia> IndexFunArray{Int}
IndexFunArray{Int64, N, F} where F where N

julia> IndexFunArray{Int} == IndexFunArray{Int64, N, F} where F where N
true

julia> IndexFunArray{Int} === IndexFunArray{Int64, N, F} where F where N
true

```

So the notation without the trailing types might be a syntactic sugar for the full notation?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [March 25, 2021, 8:44am UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/11 "2021-03-25T08:44:39Z")

</div>

You’re essentially right about both things 🙂

Having said that, let me add that, in some sense, struct signatures differ from function signatures here a bit

```julia
julia> struct MyStruct{C<:AbstractArray{T}} where T end
ERROR: syntax: invalid type signature around REPL[4]:1
Stacktrace:
 [1] top-level scope at REPL[4]:1

julia> f(x::AbstractArray{T}) where T = "whatever" # works
f (generic function with 1 method)

julia> struct MyStruct{C<:AbstractArray{T} where T} end # works

julia> struct MyStruct{T, C<:AbstractArray{T}} end # works

```

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [March 25, 2021, 9:00am UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/12 "2021-03-25T09:00:36Z")

</div>

```julia
julia> struct MyStruct{T<:Real} end # does what you want

julia> struct MyStruct{T} where {T<:Real} end # doesn't work
ERROR: syntax: invalid type signature around REPL[3]:1
Stacktrace:
 [1] top-level scope at REPL[3]:1

julia> struct MyStruct{T} <: Any where {T<:Real} end # doesn't do what you want

julia> MyStruct{ComplexF64}
MyStruct{Complex{Float64}}

```

Interesting 😃

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 25, 2021, 11:29am UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/13 "2021-03-25T11:29:37Z")

</div>

@roflmaostc, thanks for the explanation, it is clear.  
Your code is very clever and too advanced for my current level but it is impressive to see that the output views not only do not allocate but their computing time does not depend on `n` in the example below?! How come? Thanks in advance for shedding some light.

_PS: I have compared it against a simple solution that broadcasts over CartesianIndices but that runs out of computer memory for large arrays. Any expert advice that may improve this situation would be appreciated too._

```julia
using IndexFunArrays, BenchmarkTools

fci(f,n,m) = f.(Tuple.(CartesianIndices((1:n,1:m)))) #broadcasting over CartesianIndices

f(x) = sum(abs2.(x))

n = 3; m = 3;
@btime fci($f,$n,$m) # 203.833 ns (3 allocations: 256 bytes)
@btime IndexFunArray(f, ($n, $m)) # 76.386 ns (2 allocations: 64 bytes)

n = 10_000_000; m = 10_000_000;
@btime fci($f,$n,$m) # ERROR: OutOfMemoryError()
@btime IndexFunArray(f, ($n, $m)) # 79.257 ns (2 allocations: 64 bytes)

```

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [March 25, 2021, 11:38am UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/14 "2021-03-25T11:38:21Z")

</div>

You run out of memory because your allocated array with `CartesianIndices` has `(10_000_000)^2` entries.

Our object is an array in some sense. When you use `arr[2, 4]` with a normal array, it fetches the entry from the memory. But with the `IndexFunArray` we really overwrite the `arr[2, 4]` to not fetch the value from the memory but instead calculate it when accessed (from the generator function and the indices `[2,4]`). So we do not allocate memory beforehand.

Here you can see that `getindex` calculate the entry value from the index position `I`:

```julia
Base.getindex(A::IndexFunArray{T,N}, I::Vararg{Int, N}) where {T,N} = return A.generator(I)

```

**Edit:**  
Maybe this minimal example helps. The documentation for defining own array types is [here](https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-array). Below an array returning 0 or 42 randomly. It can be of arbitrary size:

```julia
julia> struct RandArray{T, N} <: AbstractArray{T, N}
           size::NTuple{N, Int}
       end

julia> Base.getindex(A::RandArray{T,N}, I::Vararg{Int, N}) where {T,N} = rand((0, 42))

julia> Base.size(A::RandArray) = A.size

julia> x = RandArray{Int, 2}((1000000000000, 1000000000000));

julia> x[1,1]
0

julia> x[1,1]
42

julia> x[1,2]
0

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 25, 2021, 12:15pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/15 "2021-03-25T12:15:59Z")

</div>

> [@roflmaostc](#):
>
> `z = IndexFunArray(x -> sum(abs2.(x)), (3, 3))`

This seems equivalent to

```julia
julia> z = mappedarray(x -> sum(abs2.(Tuple(x))), CartesianIndices((3,3)))
3×3 mappedarray(var"#3#4"(), ::CartesianIndices{2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) with eltype Int64:
  2 5 10
  5 8 13
 10 13 18

julia> z[3,3]
18

```

If you want similar syntax to your `IndexFunArray`, you could define:

```julia
indexfunarray(f, dims) = mappedarray(x -> f(Tuple(x)), CartesianIndices(dims))

```

> [@roflmaostc](#):
>
> One then could use [MappedArrays.jl](https://github.com/JuliaArrays/MappedArrays.jl) but which does need an beforehand allocated array as input

`CartesianIndices` is like a range object—it doesn’t allocate.

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [March 25, 2021, 1:05pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/16 "2021-03-25T13:05:22Z")

</div>

That makes totally sense, thanks for pointing that out!

We probably fix that in the future and get rid of `IndexFunArray` type completely 🙂

---

<div class="post-metadata">

**Author:** ![cgeoga](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cgeoga/32/216186_2.png) [@cgeoga](https://discourse.julialang.org/u/cgeoga)\
**Post date:** [March 25, 2021, 1:09pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/17 "2021-03-25T13:09:17Z")

</div>

Another similar package is [KernelMatrices.jl](https://bitbucket.org/cgeoga/kernelmatrices.jl/src/master/). Not exactly the same, because it’s really designed for passing in something like `pts::Vector{SVector{D,T}}` and then working lazily with arrays `A[j,k] = K(pts[j], pts[k], ...)` in an efficient way. So not O(1) allocations like it looks like this is designed for, but certainly similar. As a disclaimer, this is my project.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 25, 2021, 1:30pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/18 "2021-03-25T13:30:45Z")

</div>

@stevengj, your solution seems to be even faster. What is the right way to benchmark it?

_PS: tried it below but the @btime result does not make sense to me (too many floating point operations in no time)…_

```julia
using BenchmarkTools

n = 10_000_000; m = 10_000_000;

using IndexFunArrays
@btime IndexFunArray($f, ($n, $m)) # 76.36 ns (2 allocations: 64 bytes)

# solution by @stevengj :
using MappedArrays
indexfunarray(f, dims) = mappedarray(x -> f(Tuple(x)), CartesianIndices(dims))

@btime indexfunarray($f, ($n,$m)) # 0.001 ns (0 allocations: 0 bytes)

```

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [March 25, 2021, 1:40pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/19 "2021-03-25T13:40:18Z")

</div>

The reason is, that you call a wrapper to create the `IndexFunArray`.  
If you remove that wrapper, `IndexFunArray` has roughly the same.

@rafael.guerra keep in mind, that you don’t calculate anything yet. You just store the information how you _can_ calculate it when accessing it via `getindex` or `[1,2]`.

You probably want to check something like `sin.(indexfunarray($f, ($n,$m))` which then really operates on all array elements, but performance is roughly the same for both methods since they do roughly the same. (in your case, the arrays are way too large to calculate and allocate `sin.(...)` explicitly)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 25, 2021, 1:53pm UTC](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862/20 "2021-03-25T13:53:59Z")

</div>

@roflmaostc, thanks a lot for the explanation, which together with the MappedArrays.jl ReadMe file, helped me understand this a bit better.  
The fact that some mapped array values are displayed in the terminal for QC, made me believe that all values were computed, which is not the case.

[Next page](https://discourse.julialang.org/t/ann-indexfunarrays-jl-fun-with-indices-and-functions-on-them/57862.md?page=2)