ChunkSplitters.jl: Simple parallel chunk splitter

General Usage
8

We released now a new version of ChunkSplitters.jl (v2.1.0) with a new and breaking interface (although the release itself is not breaking because the previous interface is still functional, although no longer documented):

Now, a typical use of chunks is of the form:

julia> x = rand(7);

julia> @threads for inds in chunks(x; n=3, split=:batch)
           @show inds
       end
inds = 1:1:3
inds = 4:1:5
inds = 6:1:7

and it returns only the indices of the array of each chunk. Additionally, the number of chunks is
set by a keyword parameter, n, and the type of splitting is set by the optional split keyword.

To retrieve the chunk index (to avoid using threadid()) enumerate should be used:

julia> @threads for (ichunk, inds) in enumerate(chunks(x; n=3))
           @show ichunk, inds
       end
(ichunk, inds) = (1, 1:1:3)
(ichunk, inds) = (2, 4:1:5)
(ichunk, inds) = (3, 6:1:7)

This last output is equivalent to the previous interface with a call of the form for (ichunk, inds) in chunks(x, 3). This last form is still supported, but will be deprecated in version 3.0 of the package.

This change makes many codes cleaner when they do not use the chunk index, and also is consistent with the general Julia syntax of using enumerate to retrieve the indices of the counter of the iteration.

Important: the new interface is currently the only documented one.

2 Likes