# Do I need to specify what kind of weights I am providing?

**URL:** <https://discourse.julialang.org/t/do-i-need-to-specify-what-kind-of-weights-i-am-providing/56806>\
**Category:** New to Julia\
**Tags:** statistics\
**Created:** [March 9, 2021, 2:03pm UTC](https://discourse.julialang.org/t/do-i-need-to-specify-what-kind-of-weights-i-am-providing/56806 "2021-03-09T14:03:27Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jo-fleck](https://avatars.discourse-cdn.com/v4/letter/j/59ef9b/32.png) [@jo-fleck](https://discourse.julialang.org/u/jo-fleck)\
**Post date:** [March 9, 2021, 2:03pm UTC](https://discourse.julialang.org/t/do-i-need-to-specify-what-kind-of-weights-i-am-providing/56806/1 "2021-03-09T14:03:27Z")

</div>

I found helpful instructions on using `weights` and its source code ([here](https://juliastats.org/StatsBase.jl/stable/weights/), [here](https://discourse.julialang.org/t/usage-of-different-types-of-weights/4780), [here](https://github.com/JuliaStats/StatsBase.jl/blob/master/src/weights.jl)) but I am still not sure I understand how to use it correctly. Maybe I am just confusing/misunderstanding something basic here…

My question: Do I need to specify what kind of weights I am supplying? If no, how does `weights` infer if I am providing frequency, analytic, or probability weights?

I am wondering about this because I need to transfrom frequency weights included in a dataset. A few of these transformed weights are less than 1. Hence, the distinction between frequency and probability or analytic weights is no longer obvious.

As a result, I am not sure if differences in weighted outcomes are due to my transformation of the weights or because `weights` assumes that the transformed weights are not of the same kind than the original ones (or both). See example below.

Many thanks for clarifying and explaining.

> using Distributions, Random, StatsBase
> 
> rng = MersenneTwister(1234);  
> N = 1000;
> 
> a\_dist = Normal(1000, 100); a = rand(rng, a\_dist, N); # vector with obs
> 
> weights\_frequency\_orig = rand(rng, 1:100, N); # vector with ‘original’ freq weights  
> weights\_frequency\_transformed = rand(rng, 0.1:0.1:100.0, N); # transformed freq weights
> 
> a\_std\_weight\_frequency\_orig = std(a, weights(weights\_frequency\_orig));  
> a\_std\_weight\_frequency\_transformed = std(a, weights(weights\_frequency\_transformed));
> 
> a\_std\_weight\_frequency\_orig == a\_std\_weight\_frequency\_transformed # false

Is this because `weights_frequency_orig != weights_frequency_transformed` or because `weights` assumes different weights?

---

<div class="post-metadata">

**Author:** ![FPGro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fpgro/32/20822_2.png) [@FPGro](https://discourse.julialang.org/u/FPGro)\
**Post date:** [March 9, 2021, 10:00pm UTC](https://discourse.julialang.org/t/do-i-need-to-specify-what-kind-of-weights-i-am-providing/56806/2 "2021-03-09T22:00:23Z")

</div>

Probably a misunderstanding, you should explicitly use `FrequencyWeights` or `fweights`, `ProbabilityWeights` or `pweights` or `AnalyticWeigths` or `aweights`. Just `weights` will not make any assumption about the type and just error out if you try to do anything that needs this information (see your third link a bit down)

---

<div class="post-metadata">

**Author:** ![maxkapur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maxkapur/32/21208_2.png) [@maxkapur](https://discourse.julialang.org/u/maxkapur)\
**Post date:** [March 10, 2021, 1:04am UTC](https://discourse.julialang.org/t/do-i-need-to-specify-what-kind-of-weights-i-am-providing/56806/3 "2021-03-10T01:04:56Z")

</div>

Am I missing something, or is the fact that you call `rand()` twice (generating independent weights for the original and “transformed” cases) the culprit here? What happens if you replace `==` in the last line with `≈`?

Notice that setting `rng` fixes the random state for the _first trial_ but it keeps updating after that:

```julia
julia> rng = MersenneTwister(1234);

julia> rand(rng)
0.5908446386657102

julia> rand(rng)
0.7667970365022592

julia> rand(rng)
0.5662374165061859

```

---

<div class="post-metadata">

**Author:** ![jo-fleck](https://avatars.discourse-cdn.com/v4/letter/j/59ef9b/32.png) [@jo-fleck](https://discourse.julialang.org/u/jo-fleck)\
**Post date:** [March 15, 2021, 1:37pm UTC](https://discourse.julialang.org/t/do-i-need-to-specify-what-kind-of-weights-i-am-providing/56806/4 "2021-03-15T13:37:24Z")

</div>

Thanks! Makes sense.

So `weights` does not make any inference about the type of weights and just forwards responses of functions in which a given type of weights leads to errors?

---

<div class="post-metadata">

**Author:** ![jo-fleck](https://avatars.discourse-cdn.com/v4/letter/j/59ef9b/32.png) [@jo-fleck](https://discourse.julialang.org/u/jo-fleck)\
**Post date:** [March 15, 2021, 1:42pm UTC](https://discourse.julialang.org/t/do-i-need-to-specify-what-kind-of-weights-i-am-providing/56806/5 "2021-03-15T13:42:59Z")

</div>

Thanks for looking into this - I realize now my illustrative example is confusing.

The weights are supposed to differ by construction: `weights_frequency_transformed ` are [0.1,100] while `weights_frequency_orig` are [1,100].

But I see your point that calling `rng` repeatedly brings in additional differences. I am not sure this makes a big difference since N = 1000 but it’s definitely confusing.

---

<div class="post-metadata">

**Author:** ![maxkapur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maxkapur/32/21208_2.png) [@maxkapur](https://discourse.julialang.org/u/maxkapur)\
**Post date:** [March 16, 2021, 12:02am UTC](https://discourse.julialang.org/t/do-i-need-to-specify-what-kind-of-weights-i-am-providing/56806/6 "2021-03-16T00:02:19Z")

</div>

Several things going on here:

- `std()` is in Statistics, not StatsBase
- Generating the weights randomly ensures that the outputs won’t be _exactly_ the same
- `0.1:0.1:100.0` should have an upper limit of `10.0` instead to be equivalent to `1:100`
- 1000 is not enough test points for the stdevs to come out close

Here is a corrected example with random weights:

```julia
using Random, Statistics # std() is in Statistics, not StatsBase

N = 100000 # More points
a = 100 .* 1000 .+ randn(N) # Equivalent to Normal(1000, 100)

weights_frequency_orig = rand(1:100, N) 
weights_frequency_transformed = rand(0.1:0.1:10.0, N) # Upper limit corrected to 10.0

a_std_weight_frequency_orig = std(a, weights(weights_frequency_orig));
a_std_weight_frequency_transformed = std(a, weights(weights_frequency_transformed));

@show a_std_weight_frequency_orig
@show a_std_weight_frequency_transformed
# a_std_weight_frequency_orig = 0.9992338257618083
# a_std_weight_frequency_transformed = 0.9968675255497309

```

And one where I actually _transform_ the original weights instead of generating new ones:

```julia
using Random, Statistics

N = 1000 
a = 100 .* 1000 .+ randn(N)

weights_frequency_orig = rand(1:100, N) 
weights_frequency_transformed = weights_frequency_orig ./ 10 # Transformed instead

a_std_weight_frequency_orig = std(a, weights(weights_frequency_orig));
a_std_weight_frequency_transformed = std(a, weights(weights_frequency_transformed));

@show a_std_weight_frequency_orig
@show a_std_weight_frequency_transformed
# a_std_weight_frequency_orig = 1.00674757512939
# a_std_weight_frequency_transformed = 1.0067475751293908

```

Notice that the result matches up to floating-point error.

To answer your original question, it isn’t clear to me which type of weights is assumed by the generic `weights()` function. For a side-by-side comparison like this, rather than hoping that `weights()` will figure out what you intended, it’d be better to use

```julia
a_std_weight_frequency_orig = std(a, ProbabilityWeights(weights_frequency_orig))
a_std_weight_frequency_transformed = std(a, ProbabilityWeights(weights_frequency_transformed))

```

or similar to remove the ambiguity, where the different weight constructors are [as listed here](https://juliastats.org/StatsBase.jl/stable/weights/).