# Calculating a standardization function from scratch

**URL:** <https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117>\
**Category:** General Usage\
**Tags:** question\
**Created:** [February 3, 2020, 2:42pm UTC](https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117 "2020-02-03T14:42:43Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![PyDataBlog](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pydatablog/32/12327_2.png) [@PyDataBlog](https://discourse.julialang.org/u/PyDataBlog)\
**Post date:** [February 3, 2020, 2:42pm UTC](https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117/1 "2020-02-03T14:42:44Z")

</div>

Hi, I am trying to create a standardization function from scratch to scale features:

Here’s my current implementation:

```julia
# Standardize training features
function scale_features(X)
    μ = mean(X, dims=1)
    σ = std(X, dims=1)

    X_norm = (X .- μ) ./ σ

    return (X_norm, μ, σ)
end

# Scale the testing features using the learned parameters
function transform_features(X, μ, σ)
    X_norm = (X .- μ) ./ σ
    return X_norm
end

MWE:
scale_features([1, -2.2, 3])

results:
([0.15249857033260472, -1.0674899923282326, 0.9149914219956281], [0.5999999999999999], [2.6229754097208002])

```

I am getting different results from pre-made functions in sklearn with Python:

```nohighlight
import numpy as np 
from sklearn.preprocessing import StandardScaler

x = np.array([1, -2.2, 3]).reshape(-1, 1)   

StandardScaler().fit_transform(x)

# results:
array([[0.18677184],
       [-1.30740289],
       [1.12063105]])

```

What am I doing wrong with my implementation? Thanks in advance.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [February 3, 2020, 2:59pm UTC](https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117/2 "2020-02-03T14:59:17Z")

</div>

Please see point 3 in [Please read: make it easier to help you](https://discourse.julialang.org/t/psa-make-it-easier-to-help-you/14757).

In other words, please also post the input data, and the “pre-made functions” you call.

---

<div class="post-metadata">

**Author:** ![ssfrr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ssfrr/32/3736_2.png) [@ssfrr](https://discourse.julialang.org/u/ssfrr)\
**Post date:** [February 3, 2020, 3:00pm UTC](https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117/3 "2020-02-03T15:00:15Z")

</div>

Can you clarify which pre-made functions you’re comparing to? A full runnable example that shows where what you’re seeing doesn’t match your expectations would be helpful.

One possibly-common gotcha is what orientation is your data in? (i.e. if you have 1000 samples of 8D data, is it a 1000x8 matrix or a 8x1000 matrix)?

---

<div class="post-metadata">

**Author:** ![PyDataBlog](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pydatablog/32/12327_2.png) [@PyDataBlog](https://discourse.julialang.org/u/PyDataBlog)\
**Post date:** [February 3, 2020, 3:09pm UTC](https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117/5 "2020-02-03T15:09:02Z")

</div>

> [@ssfrr](#):
>
> Can you clarify which pre-made functions you’re comparing to? A full runnable example that shows where what you’re seeing doesn’t match your expectations would be helpful.

Updated the question with MWE.

---

<div class="post-metadata">

**Author:** ![ssfrr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ssfrr/32/3736_2.png) [@ssfrr](https://discourse.julialang.org/u/ssfrr)\
**Post date:** [February 3, 2020, 3:25pm UTC](https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117/6 "2020-02-03T15:25:52Z")

</div>

From the docs of `std`:

> The algorithm returns an estimator of the generative distribution’s standard deviation under the assumption that each entry of itr is an IID drawn from that generative distribution. For arrays, this computation is equivalent to calculating `sqrt(sum((itr .- mean(itr)).^2) / (length(itr) - 1))`. If `corrected` is `true`, then the sum is scaled with `n-1`, whereas the sum is scaled with `n` if corrected is false with `n` the number of elements in `itr`.

`std` by default does a “corrected” standard deviation. This is what you want if the mean of your data is estimated by taking the average of the data. If you know the mean a-priori and subtract it off manually or provide it with the `mean` argument, then you should give `corrected=false` to `std`.

So I think your implementation is doing the right thing and `sklearn` is not.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 3, 2020, 3:55pm UTC](https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117/7 "2020-02-03T15:55:54Z")

</div>

> [@ssfrr](#):
>
> So I think your implementation is doing the right thing and `sklearn` is not.

Standardization is just a simple procedure with two major benefits: easier interpretation and better numerical properties (of the likelihood etc, it’s a simple preconditioning).

There is no single canonical way to do it, either using the uncorrected or the corrected `std` is fine, as long as it is used consistently.

There are also alternative approaches, eg using [two standard deviations](http://www.stat.columbia.edu/~gelman/research/published/standardizing7.pdf). Which are also fine.

---

<div class="post-metadata">

**Author:** ![ssfrr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ssfrr/32/3736_2.png) [@ssfrr](https://discourse.julialang.org/u/ssfrr)\
**Post date:** [February 3, 2020, 4:18pm UTC](https://discourse.julialang.org/t/calculating-a-standardization-function-from-scratch/34117/8 "2020-02-03T16:18:47Z")

</div>

> [@Tamas\_Papp](#):
>
> There is no single canonical way to do it, either using the uncorrected or the corrected `std` is fine, as long as it is used consistently.

That’s fair, saying the `sklearn` is doing the wrong thing was overly harsh. Also it matters less and less as you get more data.