# Alternative to mapslices that does not allocate slices

**URL:** <https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820>\
**Category:** General Usage\
**Tags:** splitapplycombine, mapslices\
**Created:** [August 21, 2018, 10:28am UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820 "2018-08-21T10:28:26Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [August 21, 2018, 10:28am UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/1 "2018-08-21T10:28:27Z")

</div>

I’m trying to understand whether there is a non-allocating equivalent of `mapslices`. I often end up needing to reduce sliced views, for example:

```julia
mapslices(sum, m, dims = n)

```

And I would like to figure out a way that does not allocate all the intermediate slices. This is somewhat trivial for `sum` as one can do `sum(m, dims = ...)` directly, but there are many cases where `mapslices` is needed. For example, if we want to sum and skip missing values, I think one would do:

```julia
mapslices(sum∘skipmissing, m, dims = n)

```

But again, I think this allocates much more than it would be necessary for this usecase.

In the case where the reduction is pairwise, for example:

```julia
mapslices(v -> reduce(+, v), m, dims = n)

```

one can obviate this with `reduce(+, m, dims = n)`, but that again doesn’t work as soon as we need to filter, for example:

```julia
mapslices(v -> reduce(+, skipmissing(v)), m, dims = n)

```

Is there some function like `mapslices` that obviates this problems?

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [September 5, 2018, 12:59pm UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/2 "2018-09-05T12:59:26Z")

</div>

Bump?

Sorry to bump this post, but in the meantime I couldn’t find an easy solution to the problem above and am really curious whether there are good alternatives to `mapslices` that I’m missing.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [September 5, 2018, 1:30pm UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/3 "2018-09-05T13:30:22Z")

</div>

I can’t help you, but I was surprised about this.

> [@piever](#):
>
> ```julia
> mapslices(sum∘skipmissing, m, dims = n)
> 
> ```

I thought that

```julia
sum(skipmissing(m), dims=n)

```

would work, and without allocations, since `sum(skipmissing(m))` works, is fast and creates no allocations. Unfortunately,

```julia
julia> sum(skipmissing(m), dims=2)
ERROR: MethodError: no method matching sum(::Base.SkipMissing{Array{Union{Missing, Float64},2}}; dims=2)

```

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [September 5, 2018, 2:17pm UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/4 "2018-09-05T14:17:04Z")

</div>

I don’t know if you are aware of this thread

> [@Bikeshedding mapslices](https://discourse.julialang.org/t/bikeshedding-mapslices/3998):
>
> I’m trying to improve the performance of mapslices, which is needed for in [https://github.com/JuliaLang/julia/issues/3893#issuecomment-304924903](https://github.com/JuliaLang/julia/issues/3893#issuecomment-304924903). The only changes I’ve made are to rename things so I understand (a little) what’s going on and some small rearrangements. I haven’t changed performance as far as I can tell. @tim.holy mentioned he might have some leads (and something about a blog post?) which is good cause I’m lost. function my\_mapslices(f, input::AbstractArray, sliced\_dimensions::Abs…

and the resulting package

> **[GitHub - bramtayl/JuliennedArrays.jl: Type stable array slicing](https://github.com/bramtayl/JuliennedArrays.jl)**
>
> Type stable array slicing. Contribute to bramtayl/JuliennedArrays.jl development by creating an account on GitHub.

should provide what you non-allocating array slices. Maybe @bramtayl can clarify?

---

<div class="post-metadata">

**Author:** ![Azamat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/azamat/32/6892_2.png) [@Azamat](https://discourse.julialang.org/u/Azamat)\
**Post date:** [September 11, 2018, 10:25pm UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/5 "2018-09-11T22:25:16Z")

</div>

There is issue for this, which after 3 years, multiple proposals and PR is still open  
[https://github.com/JuliaLang/julia/issues/14491](https://github.com/JuliaLang/julia/issues/14491)

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [September 12, 2018, 6:06am UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/6 "2018-09-12T06:06:29Z")

</div>

I like [MLDataUtils.jl](https://github.com/JuliaML/MLDataUtils.jl) for this  
It exposes `eachobs` and `obsview` for iterators over different dimensions.

`obview` is a lazy view and allocates nothing (and is unsafe to run, e.g. `collect` on),  
while `eachobs` allocates a buffer that it reuses.

You can see below that it gives basically the same number of allocations for all the operations.

```julia
julia> using MLDataUtils
julia> using BenchmarkTools

julia> m = [rand() > 0.5 ? missing : rand() for ii in 1:10, jj in 1:20, kk in 1:30];

############ sum

julia> @btime sum($m, dims = 2);
  58.299 μs (13 allocations: 3.17 KiB)

julia> @btime mapslices(sum, $m, dims = 2);
  1.070 ms (3044 allocations: 166.47 KiB)
  
julia> @btime map(sum, eachobs($m, ObsDim.Constant{2}()));
  136.505 μs (166 allocations: 8.64 KiB)

julia> @btime map(sum, obsview($m, ObsDim.Constant{2}()));
  70.155 μs (101 allocations: 3.22 KiB)
  
############ sum∘skipmissing

julia> @btime mapslices(sum∘skipmissing, $m, dims = 2);
  1.262 ms (4543 allocations: 108.23 KiB)

julia> @btime map(sum∘skipmissing, eachobs($m, ObsDim.Constant{2}()));
  137.606 μs (246 allocations: 10.05 KiB)

julia> @btime map(sum∘skipmissing, obsview($m, ObsDim.Constant{2}()));
  67.717 μs (43 allocations: 1.83 KiB)

############ v -> reduce(+, skipmissing(v))

julia> @btime mapslices(v -> reduce(+, skipmissing(v)), $m, dims = 2);
  1.285 ms (4543 allocations: 108.23 KiB)

julia> @btime map(v -> reduce(+, skipmissing(v)), eachobs($m, ObsDim.Constant{2}()));
  139.516 μs (246 allocations: 10.05 KiB)

julia> @btime map(v -> reduce(+, skipmissing(v)), obsview($m, ObsDim.Constant{2}()));
  67.685 μs (43 allocations: 1.83 KiB)

```

I believe [JuliennedArrays.jl](https://github.com/bramtayl/JuliennedArrays.jl),  
mentioned above;  
and possibly [SplitApplyCombine.jl](https://github.com/JuliaData/SplitApplyCombine.jl)  
can also do this.  
However, I am less familiar with them.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 12, 2018, 7:43am UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/7 "2018-09-12T07:43:48Z")

</div>

For the specific case of summing while skipping missing values, see my PR:  
[https://github.com/JuliaLang/julia/pull/28027](https://github.com/JuliaLang/julia/pull/28027)

Regarding the need for a view-based `mapslices`, I totally agree, since for most operations you don’t want to mutate the inputs. I suggest filing an issue, since the existing ones are not exactly about that.

---

<div class="post-metadata">

**Author:** ![andyferris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andyferris/32/235_2.png) [@andyferris](https://discourse.julialang.org/u/andyferris)\
**Post date:** [September 12, 2018, 10:37am UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/8 "2018-09-12T10:37:05Z")

</div>

You could try the `SplitApplyCombine` package - which was designed for pulling apart data and doing operations on subsets.

In this case you might be interested in `splitdimsview`?

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [September 12, 2018, 1:28pm UTC](https://discourse.julialang.org/t/alternative-to-mapslices-that-does-not-allocate-slices/13820/9 "2018-09-12T13:28:09Z")

</div>

> I like [MLDataUtils.jl](https://github.com/JuliaML/MLDataUtils.jl) for this

That definitely looks interesting, I’ll go over the package to see what other “goodies” I can take from there. The downside of keeping this there is that the package has “low discoverability” (I’d never think of looking in what seems to be a low-level package for machine learning as I’m not a machine learning practitioner).

> I believe [JuliennedArrays.jl](https://github.com/bramtayl/JuliennedArrays.jl),  
> mentioned above;  
> and possibly [SplitApplyCombine.jl](https://github.com/JuliaData/SplitApplyCombine.jl)

JuliennedArrays is probably the closest to what I’m after, as suggested above I’ve opened an [issue](https://github.com/JuliaLang/julia/issues/29146) in Julia Base to propose to port some of the functionality in Base to go hand in hand with the already existing allocating `mapslices`.