Way to efficiently view lists of lists as a read-only flattened list?

General Usage
1

Let’s say you have a list of lists that you want to view as a single list:

super_list = [
    [ 1 , 2 , ... , 1001234 ],
    [ 6 , 1 , ... , 3 ],
    ...
    [ 42 ]
]

# _squash is some function that does read-only vcat'ing
squashed_list = _squash(super_list)

@assert in(42, squashed_list)

What is an efficient way to make a view over these lists, with indexing as if they were all vcat’ed together?

// this treats the squashed_list as a read-only array

2

Iterators.flatten(list) should be your solution, it does not allocate.

If you need indexing you could use IterTools.nth function over the flattened vector.

3

It looks like Iterators.jl is deprecated, and that functionality isn’t replaced. So I dont think this resolves the issue.

4

It got deprecated because a lot of the functionality moved into Base. See here for the flatten function.

The old packages README doesn’t really make that clear, maybe a PR indicating the change would be good.

5

Thanks. I was confused because neither flatten nor Iterators are exported for me. I have to use using Base.Iterators to get it. That’s expected, right?

6

No, the module Iterators is in the namespace, so just use Iterators.flatten directly.

7

thanks. I’ll look into what’s going on, then.

8

Like @djsegal, I’ve been struggling to find an efficient idiomatic way to reduce a function over the individual elements of an Array{Array{T}}. Even when the reducing function doesn’t need to allocate, say something like max, I still see quite a bit of allocations.

x = [rand(100) for i in 1:50]
# assuming f itself doesn't allocate - e.g. max
f1(x) = reduce(max, Iterators.flatten(x))
@benchmark f1($x)
#  memory estimate:  312.52 KiB
#  allocs estimate:  10001

f2(x) = reduce(max, (z for y=x for z=y))
@benchmark f2($x)
#  memory estimate:  314.88 KiB
#  allocs estimate:  10102

I would appreciate it if anyone could suggest a way to achieve this without allocations. It would be a shame to have to write a for loop.