ANN: MaxMinFilters.jl - fast streaming maximum / minimum within moving window

[Jakub_Wronowski]

Nice summary, I can see you were working a lot on this algorithm.

@sairus7

Steteless functions (to process small data chunks at once) => stateful functions for (realtime or larger than memory) data streams, that can be stopped and resumed later, so function state can be saved and loaded. This is also related to delays and boundary conditions handling.

This is the most interesting for me - processing of very large or infinite streams from databases or other “larger than memory” sources. I don’t think stopping and resuming is crucial but probably you have more insight in this concept than me, so maybe just I cannot imagine a use case. There are coroutines, tasks and threading (see ThreadPools.jl), so they can handle interrupts in the data delivery quite nice.

@sairus7

Special function types:

Those functionalities sounds complicated, I am not sure the O(N) algorithm will handle them.

@sairus7

Functions for 1-dimensional signals => multi-diomensional signals, like video streams.

I guess we neeed an array of Deques to handle that. Probably it would be useful somewhere but I have only 1D cases to handle.

@sairus7

Speaking of transducers, I just don’t fully understand, how to use them for complex processing chains, if you need branching and joining data streams, select to store some intermediate data, log intermediate data for debug purpose, and so on.

I don’t think they can handle that. Branching streams is not easy. If we have an iterator to database query stream it would be problematic to iterate it in 2 places with different cadence. The solution would be to open 2 independent connections and process them separately, eventually we can join those 2 processed streams later.

@sairus7

Also, I think we should start with as simple use cases as possible.

So for me simplest use case is to make this fast extrema algorithm work for any iterable things, not only AbstractArrays. I’ll try to prototype it soon, the discussion you linked is helpful. I did it long time ago in C# so this is feasible.