It looks like the problem is with extrema function. I put details in existing github issue. It describes different problem but they can be fixed together. I think I have draft of the solution, if no one will reply I will open pull request.
Flatten is a typical example where performance is not great with the lazy iterator approach. Fortunately, we’ll have transducers in Julia 1.4. It means that we can write efficient “fused” extrema quite easily:
julia> using BenchmarkTools
julia> @btime extrema(Iterators.flatten(($signal1, $signal2)));
258.958 ms (20000002 allocations: 610.35 MiB)
julia> @btime mapfoldl(
x -> (x, x),
((a, b), (c, d)) -> (min(a, c), max(b, d)),
42.014 ms (0 allocations: 0 bytes)
OK, I lied. Above example actually does not work yet, until this bug fix is merged
Trying to comment here actually helped me find out this bug
The ugly mapfoldl I wrote above is just a plain old mapfoldl. It should be executable in all Julia 1.x.
I should’ve clarified that transducers built in Base is just an implementation detail for executing foldl (see https://github.com/JuliaLang/julia/pull/33526). Things like sum(x for x in xs if x !== missing) are processed through transducers “behind your back” so that it can be compiled to a better machine code. But there is no stable API for touching transducers in Base.
I think it’s better to use transducers in reduce as well so that you can just write extrema(Iterators.flatten(...)) and it’s fast.
@tkf, thank you so much, I am happy my example helped you to fix that
I am coming from C# and F# world, and I am really missing those pipelining features. When I first realised that in Julia map(x->2x, someArray) allocates and array for its result I was more than disappointed