# Is it possible to index into a set of columns of a 3D array in a single line?

**URL:** <https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695>\
**Category:** New to Julia\
**Tags:** indexing, matrices\
**Created:** [February 3, 2022, 1:14am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695 "2022-02-03T01:14:15Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [February 3, 2022, 1:14am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/1 "2022-02-03T01:14:15Z")

</div>

if I have a 3D matrix:  
foo = rand(10,10,100)

and i want to extract:

foo[1,2,:] and foo[3,4,:]

with a single index, is there a way to do that? I don’t want to use a loop as foo in my case is chunked data that lives in the cloud so multiple calls would result in unnecessary overhead.

I have searched far and wide but can’t seem to find what I’m looking for. Any insights would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![tomerarnon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomerarnon/32/3170_2.png) [@tomerarnon](https://discourse.julialang.org/u/tomerarnon)\
**Post date:** [February 3, 2022, 1:32am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/2 "2022-02-03T01:32:11Z")

</div>

If I understand what you mean, then the answer is no. You can do this in one “call” easily enough:

```julia
col1, col2 = getindex.(Ref(foo), (1,3), (2,4), :)

```

But the broadcast makes it so it’s essentially doing the loop over the x-y pairs (1,2) and (3,4).

---

<div class="post-metadata">

**Author:** ![gustaphe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gustaphe/32/18174_2.png) [@gustaphe](https://discourse.julialang.org/u/gustaphe)\
**Post date:** [February 3, 2022, 5:19am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/3 "2022-02-03T05:19:52Z")

</div>

In benchmarks I can’t see a big difference between this and the twoliner. Seems the compiler figures it out. That holds even if I make the column longer and swap the indices so there _should_ be a big advantage to running along the long axis first.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [February 3, 2022, 8:05am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/4 "2022-02-03T08:05:41Z")

</div>

Another option would be

```julia
foo[[CartesianIndex((1,2)),CartesianIndex((3,4))],:]

```

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [February 3, 2022, 8:10am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/5 "2022-02-03T08:10:10Z")

</div>

> [@alex-s-gardner](#):
>
> I don’t want to use a loop as foo in my case is chunked data that lives in the cloud so multiple calls would result in unnecessary overhead.

Ok, I just saw this comment, with which package did you open the array? Different packages like HDF5, NCDatasets, Zarr etc have their own `getindex` implementations and therefore the ideal way to read this would depend on the package you are using.

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [February 3, 2022, 8:14am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/6 "2022-02-03T08:14:57Z")

</div>

Perhaps more concisely as

```julia
julia> col1, col2 = [foo[i...,:] for i in ((1,2),(3,4))];

julia> col1 == foo[1,2,:]
true

julia> col2 == foo[3,4,:]
true

```

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [February 3, 2022, 8:16am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/7 "2022-02-03T08:16:45Z")

</div>

Again, this comprehension is doing multiple calls to `getindex`, which is what the OP wants to avoid.

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [February 3, 2022, 5:10pm UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/8 "2022-02-03T17:10:20Z")

</div>

@fabiangans thanks a ton for the examples. I’m working with Zarr.jl and want to ensure that I’m not uncompromising a chunk more than once when pulling discrete columns of data. I’ll do some more digging/testing to see which stagey works best.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [February 3, 2022, 5:17pm UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/9 "2022-02-03T17:17:39Z")

</div>

@alex-s-gardner I did some tests as well and currently none of the suggested solutions will work the way you intend to. The array indexing in Zarr.jl is done through [https://github.com/meggart/DiskArrays.jl](https://github.com/meggart/DiskArrays.jl) , it would be good if you opened an issue there, fixing this would automatically help other packages based on DiskArrays like NetCDF.jl and ArchGDAL.jl

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [February 3, 2022, 5:25pm UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/10 "2022-02-03T17:25:45Z")

</div>

@fabiangans thanks for doing that… I’ll open an issue in DiskArrays.JL

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 4, 2022, 12:09am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/11 "2022-02-04T00:09:10Z")

</div>

> [@fabiangans](#):
>
> ```julia
> foo[[CartesianIndex((1,2)),CartesianIndex((3,4))],:]
> 
> ```

Could we make it more compact? Say:

```julia
foo[[CartesianIndex.((1,3),(2,4))...],:]

```

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [February 4, 2022, 12:59am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/12 "2022-02-04T00:59:43Z")

</div>

I like it… thanks… though the only option I could figure out for DiskArrays is to create a binary index but it’s amazingly inefficient.

In my case specific case:

```julia
mask = falses(size(foo["var"]))
mask[1, 1, :] .= true 
foo["var"][mask]

```

takes 30 seconds to read in and:

`foo["var"][1,1,:]`  
takes 0.5 seconds to read

I’ve posted on DiskArrays

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [February 4, 2022, 9:29am UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/13 "2022-02-04T09:29:50Z")

</div>

> [@rafael.guerra](#):
>
> > [@fabiangans](#):
> >
> > ````julia
> > > foo[[CartesianIndex((1,2)),CartesianIndex((3,4))],:]
> > > ```
> > 
> > ````
> 
> Could we make it more compact? Say:
> 
> ```julia
> foo[[CartesianIndex.((1,3),(2,4))...],:]
> 
> ```

Each version can be made a bit more compact:

```julia
# Using two scalars instead of a tuple in the CartesianIndex constructor:
foo[[CartesianIndex(1,2), CartesianIndex(3,4)], :]

# Broadcasting with array arguments to get result as array directly:
foo[CartesianIndex.([1,3], [2,4]), :]

```

---

<div class="post-metadata">

**Author:** ![Fliks](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fliks/32/2494_2.png) [@Fliks](https://discourse.julialang.org/u/Fliks)\
**Post date:** [March 2, 2022, 11:27pm UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/15 "2022-03-02T23:27:35Z")

</div>

You could use YAXArrays.jl for that, but I am not sure, whether this is pushing a square peg through a round hole.

```julia
using YAXArrays

function innerapplymask(xout, xin, xmask, outvec)
    if only(xmask)
        push!(outvec, deepcopy(xin))
        return nothing
    else
        return nothing
    end
end

function outerapplymask(cube, mask)

    maskaxs = caxes(mask)
    applyaxs = setdiff(caxes(cube), caxes(mask))
    indims = InDims(applyaxs..., window_oob_value=-99, artype=YAXArray)
    #inmask = InDims(MovingWindow.(maskaxs, 0,0)..., window_oob_value=0)
    outdims=OutDims()
    outvec = []
    mapCube(innerapplymask, (cube, mask), outvec; indims=(indims, InDims()), outdims)
    return outvec
end

foo=rand(10,10,100)
maskarr = falses(10,10)
maskarr[1,2] = true
maskarr[3,4] = true
outerapplymask(YAXArray(foo), YAXArray(maskarr))

```

This gives you a list of YAXArray but you could change the artype in the `InDims` constructor to `Array` to get a list of plain arrays.

For my example YAXArray with a Zarr backend of this size:

```julia
YAXArray with the following dimensions
Lon Axis with 465 Elements from 672639.15 to 686559.15
Lat Axis with 444 Elements from 9.45343066e6 to 9.44014066e6
Time Axis with 100 Elements from 2016-10-03T10:12:28 to 2020-02-09T10:12:44
Polarisation Axis with 2 elements: VH VV 
name: layer
Total size: 157.52 MB

```

this takes  
2.401743 seconds (2.15 M allocations: 859.379 MiB, 60.18% gc time)  
compared to the list comprehension:

```julia
julia> @time [freqcube[ind.I...,:,1] for ind in CartesianIndices(mask.data) if mask.data[ind]]
1361.023080 seconds (3.35 M allocations: 1.973 TiB, 17.06% gc time, 0.01% compilation time)

```

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [March 4, 2022, 8:22pm UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/16 "2022-03-04T20:22:56Z")

</div>

Hey Felix, thanks for the great example. I’ll test implementing it in my workflow.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [June 13, 2022, 3:52pm UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/17 "2022-06-13T15:52:08Z")

</div>

> [@fabiangans](#):
>
> Another option would be
> 
> ```julia
> foo[[CartesianIndex(1,2),CartesianIndex(3,4)],:]
> 
> ```

Just to reference, now that [Batch getindex by meggart · Pull Request #59 · meggart/DiskArrays.jl · GitHub](https://github.com/meggart/DiskArrays.jl/pull/59) got merged and tagged, this is optimized in all packages that use DiskArrays.jl for indexing and should be the recommended way to extract a set of columns as you describe for Zarr.jl.

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [September 3, 2022, 6:11pm UTC](https://discourse.julialang.org/t/is-it-possible-to-index-into-a-set-of-columns-of-a-3d-array-in-a-single-line/75695/18 "2022-09-03T18:11:56Z")

</div>

This has now been solved with [DiskArrays.jl](https://github.com/meggart/DiskArrays.jl) merge [#59](https://github.com/meggart/DiskArrays.jl/pull/59)