I really understand your pain:
Hi, for a library I am working now I need to find minimum and maximum values. I am looking elegant way to express that.
signal1 = rand(10000000)
signal2 = rand(10000000)
@time lo, hi = extrema(vcat(signal1, signal2))
0.159702 seconds (7 allocations: 152.588 MiB)
Code above looks good but allocates lots of memory just to iterate values, so it’s not making sense for me.
In C# I can write
(lo, hi) = signal1.Concat(signal2).Extrema();
and there is no allocation at all. It there any predefined…
opened 12:27PM - 15 Jan 20 UTC
closed 07:29AM - 05 Jul 20 UTC
fold
Hi,
The issue I am reporting is related to discourse conversation: https://di… scourse.julialang.org/t/concatenating-iterables-without-allocating-memory/33282
Also I commented some details to other issue: https://github.com/JuliaLang/julia/issues/31442#issuecomment-573939429
I encountered it using `extrema` function with flattened iterator, but here I am providing more distilled example.
First of all I need 2 big arrays which we are going to concatenate with a lazy iterator.
```
const signal1 = rand(10000000)
const signal2 = rand(10000000)
const flat = Iterators.flatten((signal1, signal2))
```
Now I am going to define a functions that will iterate this flattened data. They are based on `extrema` implementation.
``` julia
function works_ok(itr)
y = iterate(itr)
(v, s) = y
y = iterate(itr, s)
while y !== nothing
(v, s) = y
y = iterate(itr, s)
end
return v
end
function gives_strange_result(itr)
y = iterate(itr)
(v, s) = y
while y !== nothing
y = iterate(itr, s)
y === nothing && break
(v, s) = y
end
return v
end
```
The second one has strange timings and memory consumption.
```
julia> @time works_ok(flat)
0.024651 seconds (7 allocations: 240 bytes)
0.9342115147070622
julia> @time gives_strange_result(flat)
0.252765 seconds (20.00 M allocations: 610.352 MiB, 22.57% gc time)
0.9342115147070622
```
There is no memory leak, I tried to run in a loop 1000 times.
If you need to add new row and still have Array, not lazy iterator, you need to move data to make place for this new bytes. Allocating new array may be perfect, just copy data, and GC will free old one.