# Extendable multi-dimensional arrays

**URL:** <https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084>\
**Category:** General Usage\
**Tags:** question\
**Created:** [September 26, 2017, 12:13pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084 "2017-09-26T12:13:08Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [September 26, 2017, 12:13pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/1 "2017-09-26T12:13:08Z")

</div>

I’m looking for a solution to extend a multi-dimensional array along the last dimension (e.g. to store incoming chunks of multi-dimensional data, when the final size is unknown). I didn’t find anything, but here’s my prototype of an `ExtendableArray` type:

```
https://gist.github.com/oschulz/6220887859daf1d363bd0073e466b5ea

```

Does something like this already exist (possibly in more refined form)? If not, are other people interested? I could turn my draft into a package.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 26, 2017, 2:20pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/2 "2017-09-26T14:20:20Z")

</div>

Looks pretty good. Because `div` is slow and `size` is performance-critical (because it’s used in bounds-checks), you might consider using a `Base.SignedMultiplicativeInverse`. If you need inspiration, see the `Base.ReshapedArray` implementation.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 26, 2017, 2:22pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/3 "2017-09-26T14:22:43Z")

</div>

Oh, and if you want to host at [JuliaArrays](https://github.com/JuliaArrays) just let me know.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [September 27, 2017, 12:17am UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/4 "2017-09-27T00:17:50Z")

</div>

Thanks for the feedback, Tim! I’ll improved it a bit, package it up, and hosting at JuliaArrays sounds fine. 🙂

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [September 27, 2017, 12:19am UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/5 "2017-09-27T00:19:20Z")

</div>

> [@tim.holy](#):
>
> div is slow and size is performance-critical (because it’s used in bounds-checks), you might consider using a Base.SignedMultiplicativeInverse

Thanks for the tip! I was worried about that div, but didn’t know about `SignedMultiplicativeInverse`.

---

<div class="post-metadata">

**Author:** ![garrison](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/garrison/32/209519_2.png) [@garrison](https://discourse.julialang.org/u/garrison)\
**Post date:** [September 27, 2017, 10:02pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/6 "2017-09-27T22:02:49Z")

</div>

I am highly interested in this. I have my own implementation of lower quality, and it would be great to see this be a package.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [September 28, 2017, 8:55am UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/7 "2017-09-28T08:55:09Z")

</div>

Thanks for your interest, @garrison - I’ll try to get on it as soon as possible (prob. next week).

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 6, 2017, 3:39pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/8 "2017-10-06T15:39:02Z")

</div>

I put up a draft here:

```
https://github.com/oschulz/ElasticArrays.jl/tree/dev

```

I didn’t want to create it under JuliaArrays directly, in case the name has to change, etc.

@tim.holy (and the other experts), I’d be glad for any comments (here via issues/line-comments). Also, do you have any advice regarding testing? I’d imagine that there’s good testing code for generic arrays around already.

---

<div class="post-metadata">

**Author:** ![garrison](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/garrison/32/209519_2.png) [@garrison](https://discourse.julialang.org/u/garrison)\
**Post date:** [October 6, 2017, 8:50pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/10 "2017-10-06T20:50:28Z")

</div>

Thank you! I’ve already integrated this into my code, and it is working perfectly.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [October 7, 2017, 11:42am UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/11 "2017-10-07T11:42:17Z")

</div>

Sounds great. Regarding testing, I don’t know of a “thoroughly test my AbstractArray type” package (not a bad idea, though). You might look at some of the other JuliaArrays packages for inspiration. Testing things like inferrability of construction (`@inferred(ElasticArray{T}(dims))`) and indexing (both `getindex` and `setindex!`), size checks, bounds checking (`@test_throws BoundsError a[0]` etc.), iteration (`for a in A ... end`), perhaps a few general utilities (`fill!`, `copy!`, etc.), and anything specific to your package (e.g., `append!` and `prepend!`) would be pretty standard.

Regarding the name, I personally think it’s fine as it is.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 7, 2017, 1:27pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/12 "2017-10-07T13:27:20Z")

</div>

Nice to hear, @garrison - please let me know if you have any suggestion on how to improve/extend it.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 7, 2017, 1:30pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/13 "2017-10-07T13:30:02Z")

</div>

A somewhat heretical question - could we allow the standard Base.Array to be resized along it’s last dimension (e.g. in 0.7)? Or is there some deeper technical issue that prevents it?

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 7, 2017, 1:30pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/14 "2017-10-07T13:30:57Z")

</div>

Thanks, I’ll get on it, then. 🙂

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [October 7, 2017, 1:34pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/15 "2017-10-07T13:34:45Z")

</div>

I think there is a deeper issue, though not having looked directly I’m not sure how serious/relevant it is. There are times where the compiler can automatically elide bounds-checks: i.e., you may not have manually added `@inbounds`, but the compiler can figure out that it would be safe and therefore adds it itself. My understanding is that this analysis becomes harder if we allow growing of anything other than 1-d arrays. But like I said, I’ve never checked that myself.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [October 7, 2017, 2:05pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/16 "2017-10-07T14:05:26Z")

</div>

You might want to click “watch” on your own repository. At least two people have submitted PRs already. Clearly you’re off to a good start 😄.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 7, 2017, 2:26pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/17 "2017-10-07T14:26:52Z")

</div>

Oops, indeed!

My, you guys were fast, will get on the PR’s tonight. 🙂

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 7, 2017, 6:27pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/18 "2017-10-07T18:27:10Z")

</div>

We do have [test/abstractarray.jl](https://github.com/JuliaLang/julia/blob/master/test/abstractarray.jl), but it’s not really designed for this purpose. It has a whole slew of `test_SOMETHING(::Type{<:AbstractArray})` functions that are designed to verify that the abstract fallbacks are working as expected.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 7, 2017, 9:49pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/19 "2017-10-07T21:49:27Z")

</div>

I have trouble with the tests, only on OS-X with Julia nightly:

> **[Travis CI - Test and Deploy with Confidence](https://app.travis-ci.com/oschulz/ElasticArrays.jl)**
>
> Travis CI enables your team to test and ship your apps with confidence. Easily sync your projects with Travis CI and you'll be testing your code in minutes.

Any hints would be very welcome.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 9, 2017, 10:44pm UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/20 "2017-10-09T22:44:53Z")

</div>

Never mind - turns out I just needed the latest Compat.jl.

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 10, 2017, 8:46am UTC](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084/21 "2017-10-10T08:46:21Z")

</div>

All done: [GitHub - JuliaArrays/ElasticArrays.jl: Resizeable multi-dimensional arrays for Julia](https://github.com/JuliaArrays/ElasticArrays.jl)

I tried to add Femtocleaner and Attobot to the repo, but I think I can’t do that myself, under ElasticArrays.  
Also, Appveyor testing currently still runs under [AppVeyor](https://ci.appveyor.com/project/oschulz/elasticarrays-jl) - who “owns” JuliaArrays on Appveyor, so that we can move testing to [AppVeyor](https://ci.appveyor.com/project/JuliaArrays/elasticarrays-jl)?

Concerning registration, any suggestions on initial version number? It seems to work, but is it a 1.0 already?

[Next page](https://discourse.julialang.org/t/extendable-multi-dimensional-arrays/6084.md?page=2)