# How does StatsBase.skewness work?
**URL:** https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146
**Category:** Data
**Created:** [January 27, 2019, 12:51pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146 "2019-01-27T12:51:12Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Gunter\_Faes](https://discourse.julialang.org/u/Gunter_Faes)
#### Post date: [January 27, 2019, 12:51pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/1 "2019-01-27T12:51:12Z")
Hello,
It is about the application of the _skewness_ function. But I admit that I am not (yet) a Julia expert. But to my question:
I have a DataFrame _Rotwein_ and when I make the following function call …
```julia
julia> StatsBase.skewness(Rotwein[:residual_sugar])
```
… I get the following error message:
```julia
julia> StatsBase.skewness(Rotwein[:residual_sugar])
ERROR: MethodError: no method matching skewness(::Array{Union{Missing, Float64},1})
Closest candidates are:
skewness(::Distributions.DiscreteUniform) at C:\Users\guent\.juliapro\packages\Distributions\WHjOk\src\univariate\discrete\discreteuniform.jl:58
skewness(::Distributions.Hypergeometric) at C:\Users\guent\.juliapro\packages\Distributions\WHjOk\src\univariate\discrete\hypergeometric.jl:61
skewness(::Distributions.EmpiricalUnivariateDistribution) at C:\Users\guent\.juliapro\packages\Distributions\WHjOk\src\empirical.jl:46
...
```
Okay, doesn’t seem to work with the DataFrame type. Now I convert this vector of the DataFrame into the vector M1:
```julia
julia> M1
1599-element Array{Union{Missing, Float64},1}:
7.4
7.8
7.8
11.2
7.4
7.4
7.9
7.3
7.8
7.5
⋮
6.3
5.4
6.3
6.8
6.2
5.9
6.3
5.9
6.0
```
and then call the function again:
```julia
julia> StatsBase.skewness(M1)
ERROR: MethodError: no method matching skewness(::Array{Union{Missing, Float64},1})
Closest candidates are:
skewness(::Distributions.DiscreteUniform) at C:\Users\guent\.juliapro\packages\Distributions\WHjOk\src\univariate\discrete\discreteuniform.jl:58
skewness(::Distributions.Hypergeometric) at C:\Users\guent\.juliapro\packages\Distributions\WHjOk\src\univariate\discrete\hypergeometric.jl:61
skewness(::Distributions.EmpiricalUnivariateDistribution) at C:\Users\guent\.juliapro\packages\Distributions\WHjOk\src\empirical.jl:46
```
And get the same error message. I don’t understand, what’s wrong? Isn’t it enough if I simply pass a vector like in R?
```julia
> library(e1071)
> skewness(Rotwein$residual.sugar)
[1] 4.53214
```
Does anyone have a clue for me?
Thank you,
Guenter
---
### Author:  [@quinnj](https://discourse.julialang.org/u/quinnj)
#### Post date: [January 27, 2019, 1:08pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/2 "2019-01-27T13:08:11Z")
The `skewness` function is defined essentially on `Vector{Real}`, but in your case, you have a `Vector{Union{Missing, Float64}}`, meaning your vector might have `missing` (null) values. You can ignore missing values by doing `skewness(skipmissing(M1))`, or you could treat your missing values in a particular way, like replacing them with 0 like `skewness(map(x->coalesce(x, 0.0), M1))`. In general, Julia has taken the approach of opting to require users to explicitly state how missing values should be handled rather than assume the user wants them handled in one way or another. That’s of course always up for debate; for example, perhaps `skewness` should just call `skipmissing` itself by default, but that would be something to open an issue in StatsBase.jl about.
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [January 27, 2019, 1:13pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/3 "2019-01-27T13:13:27Z")
See
[https://github.com/JuliaStats/StatsBase.jl/issues/449](https://github.com/JuliaStats/StatsBase.jl/issues/449)
I think `skewness(skipmissing(M1))` won’t work, you need `skewness(collect(skipmissing(M1)))`.
---
### Author:  [@Gunter\_Faes](https://discourse.julialang.org/u/Gunter_Faes)
#### Post date: [January 27, 2019, 1:52pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/4 "2019-01-27T13:52:59Z")
Thanks for pointing out that “Missing values” was displayed. The whole record “red wine” contains no “missing values” :
The hint from _nalimilan_ was helpful (see there), only that doesn’t make _sense_ either, because no values are missing…!?
I’m still confused about the function behavior, but with this hint it works. 😉
Thank you,
Guenter
---
### Author:  [@Gunter\_Faes](https://discourse.julialang.org/u/Gunter_Faes)
#### Post date: [January 27, 2019, 2:00pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/5 "2019-01-27T14:00:01Z")
Thanks, that’s how it worked, even with DataFrames! Only I don’t understand it because there are no missing values marked in the vector…! But here is the working example:
```julia
julia> M1 = collect(skipmissing(Rotwein[:residual_sugar]))
1599-element Array{Float64,1}:
1.9
2.6
2.3
1.9
1.9
1.8
1.6
1.2
2.0
```
As expected, there are still 1599 observations. But now the function call works:
```julia
julia> StatsBase.skewness(M1)
4.536394788805635
```
Also available as DataFrame:
```julia
julia> StatsBase.skewness(collect(skipmissing(Rotwein[:residual_sugar])))
4.536394788805635
```
R does not detect missing values where there are none. As I said, I am a little surprised.
Thank you,
Guenter
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [January 27, 2019, 2:40pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/6 "2019-01-27T14:40:51Z")
> [@quinnj](#):
>
> perhaps `skewness` should just call `skipmissing` itself by default
I don’t think this would be a good idea. The next request would be that it also calls `skipnothing`, then `filter(!isnan)`, etc. I think it is much better to expect valid input from the user, but at the same time make it easy to satisfy this requirement. `skewness(skipmissing(...))` is much more informative to read because I know precisely what is going on.
---
### Author:  [@kevbonham](https://discourse.julialang.org/u/kevbonham)
#### Post date: [January 27, 2019, 3:37pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/7 "2019-01-27T15:37:36Z")
> [@Gunter\_Faes](#):
>
> Thanks, that’s how it worked, even with DataFrames! Only I don’t understand it because there are no missing values marked in the vector…!
The function doesn’t know there are no missing values, it’s going based off the type of the vector, which you can see in the error message. Not sure where that type signature came from, but if it really has no missing, you can replace the column with a pure Float vector
---
### Author:  [@Gunter\_Faes](https://discourse.julialang.org/u/Gunter_Faes)
#### Post date: [January 27, 2019, 3:46pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/8 "2019-01-27T15:46:08Z")
Perhaps we should think about the idea of ignoring missing values in the argument? What contribution do missing values have to a distribution? It is more important to have a sufficient number of observations to estimate the skewness. Then a hint that too few observations are available would be more helpful. Or not?
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [January 27, 2019, 3:54pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/9 "2019-01-27T15:54:54Z")
> [@Gunter\_Faes](#):
>
> What contribution do missing values have to a distribution?
This question can have very complex answers, depending on how they ended up missing, this is probably beyond the scope of `StatsBase`. I was simply making the point that automatically applying `skipmissing` would
1. is tantamount to [MCAR](https://en.wikipedia.org/wiki/Missing_data#Missing_completely_at_random), implicitly making a choice that should be made by the user,
2. also masks bugs in the caller’s code (if `missing` values were unexpected),
3. opens the door for requests for other kinds of input sanitization.
---
### Author:  [@kevbonham](https://discourse.julialang.org/u/kevbonham)
#### Post date: [January 27, 2019, 3:58pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/10 "2019-01-27T15:58:54Z")
> [@Gunter\_Faes](#):
>
> Perhaps we should think about the idea of ignoring missing values in the argument? What contribution do missing values have to a distribution? It is more important to have a sufficient number of observations to estimate the skewness.
Maybe, but that’s a decision one should make explicitly. If one has missing values, the distribution might be quite different than what you would think based on the observed values.
---
### Author:  [@matthieu](https://discourse.julialang.org/u/matthieu)
#### Post date: [January 27, 2019, 5:07pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/11 "2019-01-27T17:07:29Z")
I would love for `skipmissing = true` to be the default. The way I see it, if a user is concerned with missing values, she can use containers that restrict the presence of missing values. I disagree that one default is more explicit than the other.
---
### Author:  [@Gunter\_Faes](https://discourse.julialang.org/u/Gunter_Faes)
#### Post date: [January 27, 2019, 5:16pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/12 "2019-01-27T17:16:26Z")
This is a good idea to provide such functions with an attribute like “missing = true/false”. Then the user also has control over the function. I can also understand the hint from _Tamas_, but in the end the functions should be user-friendly and not too nested.
> [@kevbonham](#):
>
> If one has missing values, the distribution might be quite different…
That may be, but it is more a problem of the number of observations. If I extend a vector (a DataFrame I leave out) that contains enough observations by missing values, I certainly don’t change the distribution. If I then exclude the missing values by _collect(skipmissing(…))_, why not use it as a function attribute? But thanks again for your answers, I just wanted to give a suggestion.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [January 28, 2019, 8:12am UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/13 "2019-01-28T08:12:43Z")
> [@Gunter\_Faes](#):
>
> provide such functions with an attribute like “missing = true/false”
You mean a keyword argument? Lazy maps like `skipmissing` are much more idiomatic in Julia, allowing composition and transformations.
---
### Author:  [@Gunter\_Faes](https://discourse.julialang.org/u/Gunter_Faes)
#### Post date: [January 28, 2019, 9:57am UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/14 "2019-01-28T09:57:28Z")
For the data analyst, dealing with missing values is a daily business. Anything that makes this easier is welcome.
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [January 28, 2019, 10:03am UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/15 "2019-01-28T10:03:55Z")
One thing that might be useful in your case:
```julia
dropmissing(Rotwein, disallowmissing = true)
```
which will drop all rows that contain `missing` - I understand in your case this doesn’t affect any columns, so the effect will essentially be to coerce all `Union{T, Missing}` columns to just being of type `T`.
You can also deal with `missing` upfront when reading in your data, e.g.
```julia
CSV.read("Rotwein.csv", allowmissing = :none)
```
Which will return only “pure” columns rather than unions. Note that this only works if there actually aren’t any `missing` values in your file, if you have some columns that contain `missing` you might want to go for `allowmissing = :auto`.
Coming from Python, R, Stata I agree that Julia’s approach to `missing` seemed rather pedantic and cumbersome at first, but I think I’ve now gotten around to this way of thinking more explicitly about the problem (and trying to purge all `Union`s where I don’t need them!)
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [January 28, 2019, 12:59pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/16 "2019-01-28T12:59:00Z")
`allowmissing = :none` or `allowmissing = :auto` is certainly a good idea for clarity, convenience and performance if you know there are no missing values. We use `allowmissing = :always` by default because parsing CSV files currently fails if missing values are not encountered during the type detection phase. This might change once CSV.jl is able to handle this situation better.
For cases where there are missing values and you want to skip them, Julia just behaves like R, except that `skipmissing` is a function while in R `na.rm` (or `useNA`, or `use="na.or.complete"`…) is an argument. That can be inconvenient when you deal with missing values all the time (I do), but so far we haven’t found any solution which would both be safe by default, but would allow opting-in for skipping missing values by default. See also [this blog post](https://julialang.org/blog/2018/06/missing) for some context and rationale.
---
### Author:  [@Gunter\_Faes](https://discourse.julialang.org/u/Gunter_Faes)
#### Post date: [January 28, 2019, 2:24pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/17 "2019-01-28T14:24:12Z")
Thanks for the clues, _nilshg_ & _nalimilan_! I will play around with the hints a little to get to know the possibilities. Of course I don’t want to lose control over the missing values. Because it is always important to know if and how many missing values are available.
I also believe - in my opinion today - that the missing values should also be read in via CSV.il, because a postprocessing in Julia leads to more knowledge about the data set.
Thanks also to all who responded, it was/is an interesting and substantial discussion!
---
### Author:  [@matthieu](https://discourse.julialang.org/u/matthieu)
#### Post date: [January 28, 2019, 3:51pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/18 "2019-01-28T15:51:19Z")
In term of adoption of Julia in the data analysis community. I can imagine a ton of people being put off by all the current problems with missing values (this issue with skewness, related issue with weighted functions).
In contrast, I can’t imagine anyone not picking up Julia because it automatically skips missing values.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [January 28, 2019, 4:33pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/19 "2019-01-28T16:33:00Z")
I think the community has become pretty immune to the _“people won’t adopt Julia unless you do what I want”_ arguments; especially as quite a few parts of various APIs were hammered out patiently through multiple (breaking) iterations instead of just picking what Matlab/R/Python does, and Julia’s adoption has kept steadily increasing.
If you want to convince people about a particular API choice, I would recommend you base your arguments on technical merits.
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [January 28, 2019, 4:41pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/20 "2019-01-28T16:41:41Z")
BTW, this is an area where improvements can be made without too much work and even without a deep experience with Julia development. One basically needs to go over all problematic functions and ensure they accept any iterator instead of just arrays. That’s a great task for a first pull request.
[Next page](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146.md?page=2)