# DiskArrays get array of values given a vector of Indices

**URL:** <https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [April 24, 2022, 9:43pm UTC](https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960 "2022-04-24T21:43:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sunayana](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sunayana](https://discourse.julialang.org/u/sunayana)\
**Post date:** [April 24, 2022, 9:43pm UTC](https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960/1 "2022-04-24T21:43:41Z")

</div>

Hello I am new to julia and currently using DiskArrays for very large matrices.  
I have a DiskArray of `volumes` of dimension `12000 x 12000` and I have a vector of `indices` `Vector{CartesianIndex{2}}` of length `47852268`. I am trying to find a way to create another DiskArray of `specific_volumes` by extracting the volumes at the specific indices, but when I try to do :

```julia
volumes.[indices]

```

I get the error : `ERROR: syntax: invalid syntax "volume.[indices]" around REPL[20]:1`.  
A simple for loop of the indices is expensive in terms of memory, is there some way of using broadcasting to solve this issue.

Thanks!

---

<div class="post-metadata">

**Author:** ![sbuercklin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sbuercklin/32/15728_2.png) [@sbuercklin](https://discourse.julialang.org/u/sbuercklin)\
**Post date:** [April 25, 2022, 2:24am UTC](https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960/2 "2022-04-25T02:24:26Z")

</div>

You could try broadcasting `getindex.(Ref(volume), indices)`. `volume[i]` is just syntactic sugar for `getindex` calls in the end.

The `Ref(volume)` tells Julia not to broadcast over `volume`, but you can read more about `Ref` [in the manual](https://docs.julialang.org/en/v1/base/c/#Core.Ref)

This likely doesn’t return another `DiskArray`, however, as broadcasting will return an `Array`

---

<div class="post-metadata">

**Author:** ![sunayana](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sunayana](https://discourse.julialang.org/u/sunayana)\
**Post date:** [April 25, 2022, 6:56am UTC](https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960/3 "2022-04-25T06:56:17Z")

</div>

Thanks for your response, unfortunately with 16Gb RAM this does not scale well and crashes Julia

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [April 25, 2022, 7:07am UTC](https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960/4 "2022-04-25T07:07:28Z")

</div>

In addition the broadcasting would be awfully slow, the whole point of DiskArrays is to avoid repeated calls to getindex. Unfortunately there is not yet a simple function that would implement your use case in DiskArrays.jl. There is a branch that supports fast batched getindex [Batch getindex by meggart · Pull Request #59 · meggart/DiskArrays.jl · GitHub](https://github.com/meggart/DiskArrays.jl/pull/59) but this will not solve your problem right now since the output is larger than RAM and the batched getindex function I implemented there writes into a normal Array.

In case you have control over your input array, have you considered using mmap for both source and target arrays?

---

<div class="post-metadata">

**Author:** ![sunayana](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sunayana](https://discourse.julialang.org/u/sunayana)\
**Post date:** [April 26, 2022, 7:44am UTC](https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960/5 "2022-04-26T07:44:19Z")

</div>

Thanks @fabiangans , since my source array is a ArchGDAL dataset when I extract a raster band I end up getting a `DiskArray`. I have tried to convert the source array from a DiskArray to a memory mapped array but I keep getting a segmentation fault for a DiskArray of type Float64 and size 12000x12000

```julia
Allocations: 30695363 (Pool: 30675019; Big: 20344); GC: 38

```

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [April 27, 2022, 8:00am UTC](https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960/6 "2022-04-27T08:00:50Z")

</div>

Ok, thanks for the clarification. In this case I think it would be best to hard-code a loop that sorts your CartesianIndex list by the chunk they belong to and then you batch-wise copy the data from one array to the other.

I will modify the batch getindex PR a bit to export some helper functions that will be useful, but I am still struggling to think of a good interface that would allow the copy operation you describe. Out of curiosity: do you know about a tool/framework in another programming language that would solve your task for arrays of this size? Just to get some inspirations for possible interfaces

---

<div class="post-metadata">

**Author:** ![sunayana](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sunayana](https://discourse.julialang.org/u/sunayana)\
**Post date:** [April 27, 2022, 11:49am UTC](https://discourse.julialang.org/t/diskarrays-get-array-of-values-given-a-vector-of-indices/79960/7 "2022-04-27T11:49:32Z")

</div>

Thanks Fabian. So this is what I have used previously:  
[https://www.boost.org/doc/libs/1\_77\_0/libs/iostreams/doc/classes/mapped\_file.html](https://www.boost.org/doc/libs/1_77_0/libs/iostreams/doc/classes/mapped_file.html)