using DataFrames data = DataFrame(time = rand(1:100, 1_000_000), val=rand(1_000_000)) consider the above data frame and say I wish to compute the average val within a rolling window and normalize each val using the rolling average and standard deviation. It’s almost like a moving average but notice…

Did you have something similar to DataFrame.rolling from pandas in mind?

While DataFrames is undoubtedly one of the most favorite ways to deal with tabular data, it is worth noting that there are other alternatives, which can be more suited for particular tasks. For example, TimeSeries.jl have special section devoted to window manipulations Apply methods · TimeSeries.jl .…

it’s a bit harder than just rolling, cos there are multiple observations per time point.

probably you cannot skip the 100 times calculation, however, you can make it faster. the groupby function gives the starts and ends of each group, so instead of using subset you can go through these starts and ends to filter observations.

A colleague showed me the Spark’s (and SQL’s?) https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.Window.rangeBetween.html which can do this. Interesting.

No. But I will check them out now

Author of RollingTimeWindows (and the associated post) here. I’m happy to give you commit access if you want to make any modifications to RollingTimeWindows (e.g. adding fixed-width iteration or functions that live at a higher level of abstraction than the iterator) for your particular use-case – ju…

How to compute the rolling standardization using DataFrames

General Usage

rafael.guerra July 21, 2021, 1:03pm 7

@xiodai, have you also taken a look at the RollingTimeWindows.jl package and associated discourse post?

Topic		Replies	Views
Calculation of rolling standard deviation by group General Usage dataframes	7	673	January 26, 2023
Normalizing DataFrame column by group Data	5	932	October 30, 2022
RollingFunctions with a variable window width New to Julia dataframes	19	1545	October 8, 2022
Creating a DataFrame column as the running mean of another column Data dataframes	5	793	January 24, 2022
Standardize all columns of DataFrame Data question , package	9	2891	October 2, 2023

How to compute the rolling standardization using DataFrames

Related topics