# Initializing OnlineStats Series with statistics

**URL:** <https://discourse.julialang.org/t/initializing-onlinestats-series-with-statistics/58792>\
**Category:** General Usage\
**Tags:** package, statistics\
**Created:** [April 7, 2021, 8:14pm UTC](https://discourse.julialang.org/t/initializing-onlinestats-series-with-statistics/58792 "2021-04-07T20:14:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jarapo](https://avatars.discourse-cdn.com/v4/letter/j/6bbea6/32.png) [@jarapo](https://discourse.julialang.org/u/jarapo)\
**Post date:** [April 7, 2021, 8:14pm UTC](https://discourse.julialang.org/t/initializing-onlinestats-series-with-statistics/58792/1 "2021-04-07T20:14:34Z")

</div>

**Premise:**  
I am running experiments that involve with Monte Carlo sampling on clusters, and I am collecting the mean and variance using `OnlineStats`.  
It takes a long time and I get time out errors on our clusters, so I want to save data and restart in another job.  
Since the number of sampling is huge, I want to store only the mean, variance and the sample size, (not the whole data) to restart.

I am aware of the algorithm [Online estimation of variance with limited memory - Cross Validated](https://stats.stackexchange.com/questions/235129/online-estimation-of-variance-with-limited-memory) (but if I were to be willing to implement this myself, I would not be using OnlineStats)

**Question:**  
If I do

`using OnlineStats`  
`mystat = Series(Mean(),Variance())`  
`fit!(mystat, rand(10))`

I get

`Series`  
`├─ Mean: n=10 | value=0.542621`  
`└─ Variance: n=10 | value=0.077484`

If I can save `mystat` and load `mystat` as a “julia variable” like matlab then that’s fine, but it seems to be tricky: [What is the preferred way to save variables? - #17 by FHell](https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918/17)

I know `value(mystat)` gives the mean and variance, and `nobs(mystat)`gives the sample size, which I can save to a `.txt` file and I can read it in another run.

But given the mean, variance, and the sample size, I don’t know how to create `Series` “mystat” with the same information, so that I can `merge!` in another run of my experiment.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [April 7, 2021, 8:37pm UTC](https://discourse.julialang.org/t/initializing-onlinestats-series-with-statistics/58792/2 "2021-04-07T20:37:48Z")

</div>

I would think as a worst case you can just rebuild the structs exactly as they were. See the definition in the code for Variance below.

[https://github.com/joshday/OnlineStatsBase.jl/blob/master/src/stats.jl#L476](https://github.com/joshday/OnlineStatsBase.jl/blob/master/src/stats.jl#L476)

---

<div class="post-metadata">

**Author:** ![jarapo](https://avatars.discourse-cdn.com/v4/letter/j/6bbea6/32.png) [@jarapo](https://discourse.julialang.org/u/jarapo)\
**Post date:** [April 7, 2021, 9:08pm UTC](https://discourse.julialang.org/t/initializing-onlinestats-series-with-statistics/58792/3 "2021-04-07T21:08:36Z")

</div>

Ah thank you!

So with  
`mystat = Series(Mean(),Variance());`  
`fit!(mystat, rand(20))`  
I get  
`Series`  
`├─ Mean: n=10 | value=0.446827`  
`└─ Variance: n=10 | value=0.0938919`

I guessed that you meant something like

`mymean = Mean(value(mystat)[1], EqualWeight() , nobs(mystat));`  
`myvar = Variance(value(mystat)[2], value(mystat)[1], EqualWeight() , nobs(mystat));`

`mypreviousstat = Series(mymean,myvar)`  
But I get the correct mean but the variance is not correct.  
`Series`  
`├─ Mean: n=10 | value=0.446827`  
`└─ Variance: n=10 | value=0.104324`

I may be misunderstanding some definitions of sample variance etc. but \sigma2 looks like the variance and I assume the definition of sample variance to be the same in the function…what did I miss…?

**Update:**  
`myvar = Variance(value(mystat)[2]*(nobs(mystat)-1)/nobs(mystat), value(mystat)[1], EqualWeight() , nobs(mystat))`

Worked. So somehow the parameter \sigma2 in the struct is the **biased** estimator of the variance (the one with 1/(sample size)) and value(mystat)[2] gives the ( **unbiased** ) sample variance (the one with 1/(sample size -1))…?

---

<div class="post-metadata">

**Author:** ![joshday](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshday/32/368_2.png) [@joshday](https://discourse.julialang.org/u/joshday)\
**Post date:** [April 11, 2021, 12:55am UTC](https://discourse.julialang.org/t/initializing-onlinestats-series-with-statistics/58792/4 "2021-04-11T00:55:01Z")

</div>

OnlineStats author here. Yes, the `Variance` struct stores the biased variance because it simplifies the update code in `fit!`.

If I’m understanding your problem correctly, you could also `serialize` the Variance in one process and then `deserialize` it in another.