# Chunk an array

**URL:** <https://discourse.julialang.org/t/chunk-an-array/101878>\
**Category:** General Usage\
**Tags:** array, iterators\
**Created:** [July 21, 2023, 10:05am UTC](https://discourse.julialang.org/t/chunk-an-array/101878 "2023-07-21T10:05:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tommy\_J](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tommy_j/32/51703_2.png) [@tommy\_J](https://discourse.julialang.org/u/tommy_J)\
**Post date:** [July 21, 2023, 10:05am UTC](https://discourse.julialang.org/t/chunk-an-array/101878/1 "2023-07-21T10:05:32Z")

</div>

Hello to all,

I would like to know how the most efficient way to chunk a multi-dimensional array in julia.

`arr = rand(Float64, (10, 100))`

How can split it into chunks of size 30 so that I get 3 (10, 30) and 1 (10,10) arrays?

I have this function:

```julia
function chunk_array(arr, N)
    chunked = []
    n_columns = size(arr)[2]

    for i in 1:N:n_columns
        push!(chunked, arr[:, i:min(i+N-1, n_columns)])
    end
    return chunked
end

```

Is there something more performant?

Best Regards

---

<div class="post-metadata">

**Author:** ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)\
**Post date:** [July 21, 2023, 10:46am UTC](https://discourse.julialang.org/t/chunk-an-array/101878/2 "2023-07-21T10:46:08Z")

</div>

You can replace `size(arr)[2]` by `size(arr, 2)` (this is more of a style improvement). You can pre-allocate the vector for the chunks.

```julia
function chunk_array_2(arr::AbstractMatrix, N::Integer)
    n_cols = size(arr, 2)
    n_chunks = ceil(Int, n_cols / N)
    chunks = Vector{typeof(arr)}(undef, n_chunks)
    for i in 1:n_chunks
        from = N * (i - 1) + 1
        to = min(i * N, n_cols)
        chunks[i] = arr[:, from:to]
    end
    return chunks
end

```

If it suits your needs, you can use a view when slicing the matrix.

```julia
function chunk_array_3(arr::AbstractMatrix, N::Integer)
    n_cols = size(arr, 2)
    n_chunks = ceil(Int, n_cols / N)
    chunks = Vector{AbstractMatrix}(undef, n_chunks)
    for i in 1:n_chunks
        from = N * (i - 1) + 1
        to = min(i * N, n_cols)
        chunks[i] = @view arr[:, from:to]
    end
    return chunks
end

```

The timings are

```julia
arr = rand(Float64, (10, 100))
@btime chunk_array($arr, 30);
  2.271 μs (6 allocations: 8.50 KiB)
@btime chunk_array_2($arr, 30);
  2.168 μs (5 allocations: 8.45 KiB)
@btime chunk_array_3($arr, 30);
  94.155 ns (5 allocations: 336 bytes)

```

---

<div class="post-metadata">

**Author:** ![xgdgsc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xgdgsc/32/608_2.png) [@xgdgsc](https://discourse.julialang.org/u/xgdgsc)\
**Post date:** [July 21, 2023, 11:11am UTC](https://discourse.julialang.org/t/chunk-an-array/101878/3 "2023-07-21T11:11:26Z")

</div>

[https://juliaml.github.io/MLUtils.jl/stable/api/#MLUtils.chunk](https://juliaml.github.io/MLUtils.jl/stable/api/#MLUtils.chunk)

---

<div class="post-metadata">

**Author:** ![mikmoore](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikmoore/32/31109_2.png) [@mikmoore](https://discourse.julialang.org/u/mikmoore)\
**Post date:** [July 21, 2023, 2:58pm UTC](https://discourse.julialang.org/t/chunk-an-array/101878/4 "2023-07-21T14:58:35Z")

</div>

Loop in Julia are fine and what you wrote is fine except for one or two things:

1. `chunked isa Vector{Any}` and will lead to poor downstream performance every time it is accessed, due to type instability. Note that `Vector{AbstractMatrix}` is basically just as bad. You want the type to be declared or inferred concretely.
2. You are making copies of the data when you write `arr[:, i:min(i+N-1, n_columns)]`. If you _want_ copies, this is fine. If it’s okay to alias the input data, consider using `@view arr[:, i:min(i+N-1, n_columns)]` instead. When aliased, changes to `arr` will be reflected in `chunked` and vice-versa, as they share memory.

I would probably write this function like this

```julia
function chunk_array(arr::AbstractVecOrMat, N)
    chunked = map(Iterators.partition(axes(arr,2),N)) do cols
        @view arr[:,cols] # remove @view if you want copies that do not alias `arr`
    end
    return chunked
end

```

---

<div class="post-metadata">

**Author:** ![tommy\_J](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tommy_j/32/51703_2.png) [@tommy\_J](https://discourse.julialang.org/u/tommy_J)\
**Post date:** [July 21, 2023, 3:04pm UTC](https://discourse.julialang.org/t/chunk-an-array/101878/5 "2023-07-21T15:04:39Z")

</div>

I actually want a copy of the data.  
Is is sufficient to declare the type of variable for efficiency?

---

<div class="post-metadata">

**Author:** ![mikmoore](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikmoore/32/31109_2.png) [@mikmoore](https://discourse.julialang.org/u/mikmoore)\
**Post date:** [July 21, 2023, 3:12pm UTC](https://discourse.julialang.org/t/chunk-an-array/101878/6 "2023-07-21T15:12:02Z")

</div>

> [@tommy\_J](#):
>
> Is is sufficient to declare the type of variable for efficiency?

Yes, if you declared the type via `chunked = THE_TYPE[]` or `chunked = Vector{THE_TYPE}(undef, num_chunks)` it would be fine. The annoying part is that the type can be a little complicated at times. But in this case, since you want the data copied, it’s pretty easy and `Matrix{eltype(arr)}` in place of `THE_TYPE` will work.

But I would still recommend you use my suggested solution with `@view` deleted. It will make the copies and let the compiler determine the type for you (thanks to `map`). Declaring types manually can be tedious (and sometimes very difficult) to do correctly.