[ANN] OnlineSampleStatistics: Online estimation of statistical moments

FerreolS
Tue, 22 Apr 2025 09:30:54 +0000
Package Announcements

Hi,
OnlineSampleStatistics.jl is a package for online single pass estimation of any statistical moments of data stream. OnlineStats.jl package was not adapted to my needs (mainly camera calibration) for two reasons that are solved in this package:

working with arrays: fast estimation mean, variance and even higher moments images from a stream of data frames,

numerically stable estimation for non centered variables: as illustrated in the README, OnlineStats suffers from catastrophic precision issues on non-centered data (that can even lead to negative variance).

OnlineSampleStatistics.jl implements two types: UnivariateStatistic for single variable and IndependentStatistic as arrays of UnivariateStatistic.

UnivariateStatistic is a subtype of OnlineStats{T} from OnlineStats.jl to leverage its functionality, including the Transducers.jl methods for parallel processing.

IndependentStatistic is a subtype of AbstractArray{UnivariateStatistic{T,K},N} that use a ZippedArrays to ensure the efficiency of the updates.

Both are fast (UnivariateStatistic is as fast as Variance() from OnlineStats), zero allocations and supports weighted data.

It is not required for my use case, but I know how to adapt it to GPUArray and Unitfull variables.

For those interested, the package is already available in the general registry. It is my first package and feedback are welcomed.

5 posts - 3 participants