# How does StatsBase.skewness work?

**URL:** <https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146>\
**Category:** Data\
**Created:** [January 27, 2019, 12:51pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146 "2019-01-27T12:51:12Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![matthieu](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@matthieu](https://discourse.julialang.org/u/matthieu)\
**Post date:** [January 28, 2019, 5:15pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/21 "2019-01-28T17:15:47Z")

</div>

I did not say “people won’t adopt Julia unless X”. I was just sharing my opinion on the subject. Thanks.

---

<div class="post-metadata">

**Author:** ![Gunter\_Faes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunter_faes/32/12580_2.png) [@Gunter\_Faes](https://discourse.julialang.org/u/Gunter_Faes)\
**Post date:** [January 28, 2019, 7:15pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/22 "2019-01-28T19:15:17Z")

</div>

Hi Tamas,  
I would like to come back to your argument (the missing value theme for me is in principle through)…

> [@Tamas\_Papp](#):
>
> If you want to convince people about a particular API choice, I would recommend you base your arguments on technical merits.

… you only convince the development community of Julia. You can only convince users who want to model mathematical problems or - like me - perform data analysis if Julia is easy to use, fast and stable. But if you say that you want to deal with missing values, you have to call this and that function in addition, the motivation is certainly not that high. Let me give you an example from another area (sorry for that): The car industry no longer advertises torque or overhead camshafts, but driving comfort and safety.

But I don’t want to stress it, I just want to give suggestions.  
Cheers,  
Guenter

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 29, 2019, 9:10am UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/23 "2019-01-29T09:10:49Z")

</div>

> [@Gunter\_Faes](#):
>
> But if you say that you want to deal with missing values, you have to call this and that function in addition, the motivation is certainly not that high.

I am not sure how you came to this conclusion — unfortunately, I don’t have reliable data on this, and extrapolating from my own preferences is not a substitute.

That said, even if there are users who would prefer a DWIM-style interface that makes some choices for you, I think it is totally fine for Julia to appeal to the people who prefer to make these choices explicit. You can’t please everyone.

Regarding missing values, automatic handling of them can be particularly insidious apart from simple cases. Consider the covariate matrix in a regression (without intercepts)

```julia
X = [1 2;
     3 4;
     5 missing;
     missing 10]

```

“Dropping missing automatically” here in OLS would presumably drop rows 3 and 4. What should

```julia
mapslices(mean, X; dims = 1)
mapslices(skewness, X; dims = 1)

```

do, drop one element from each column automatically? That would make it inconsistent with other calculations. The small price you pay for being explicit about how you handle missing values saves you from a lot of potential bugs.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [January 29, 2019, 11:43am UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/24 "2019-01-29T11:43:53Z")

</div>

> [@Tamas\_Papp](#):
>
> That said, even if there are users who would prefer a DWIM-style interface that makes some choices for you, I think it is totally fine for Julia to appeal to the people who prefer to make these choices explicit. You can’t please everyone.

Alternatively, someone is free to write a package with more DWIM and users that like Julia but prefer that Melanie are welcome to use it. I can certainly see the value, and might even want that sort of behavior in some circumstances. But I definitely think the default behavior should require explicitness.

---

<div class="post-metadata">

**Author:** ![matthieu](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@matthieu](https://discourse.julialang.org/u/matthieu)\
**Post date:** [January 29, 2019, 6:43pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/25 "2019-01-29T18:43:53Z")

</div>

I guess one could create a wrapper type, `SkipMissingArray <: AbstractArray`. While `skipmissing` returns an iterator, `skipmissingarray` returns a `SkipMissingArray`.

It would solve the weighted average issue, see [https://discourse.julialang.org/t/how-to-calculate-a-weighted-mean-with-missing-observations/1928](https://discourse.julialang.org/t/how-to-calculate-a-weighted-mean-with-missing-observations/1928), because one could define a particular method `mean(x::SkipMissingArray, w::Weights(<:SkipMissingArray)`

One issue is that functions tend to output `Vector{Union{Missing, Array}}` so users would always need to convert every created vector into a `SkipMissingArray`. It would be very cumbersome. I’m not sure one can write a package that acts as if the default behavior of Julia was to automatically skip missings.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 29, 2019, 7:29pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/26 "2019-01-29T19:29:07Z")

</div>

One problem with that approach is that `skipmissing` is very memory efficient because you can’t index a `SkipMissing` object. (You can’t do `skipmissing(x)[4]` for example). This is a trade-off that makes sense because as stated above, the vast majority of functions don’t _really_ need an array input. Actually creating an object that has all the behaviors required for an `Array` likely requires looking at the original object in a less efficient way.

The issue you linked to isn’t quite the same. I am more there about making sure the iterators are synced, without necessarily allowing indexing of each one.

---

<div class="post-metadata">

**Author:** ![matthieu](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@matthieu](https://discourse.julialang.org/u/matthieu)\
**Post date:** [January 29, 2019, 8:18pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/27 "2019-01-29T20:18:11Z")

</div>

In case where the iterator `skipmissing` is enough, then it would just use it, i.e.:

```julia
mean(x::SkipMissingArray) = mean(skipmissing(x))

```

The key point is that the result of `skipmissingarray` is an AbstractArray, so that, from the user point of view, it still behaves as a normal array, e.g. one can have a DataFrames of SkipMissingArrays, etc.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [January 29, 2019, 8:36pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/28 "2019-01-29T20:36:35Z")

</div>

> [@matthieu](#):
>
> one can have a DataFrames of SkipMissingArrays, etc.

That would only be possible if all columns had the same number of missing values, or they wouldn’t have the same length.

Anyway `skipmissing` already returns a `SkipMissing` object for which special methods can be defined (which is already used a lot for reductions in Base). As [I noted](https://discourse.julialang.org/t/how-to-calculate-a-weighted-mean-with-missing-observations/19281/9) in the thread about the weighted mean, this could perfectly be used there.

---

<div class="post-metadata">

**Author:** ![matthieu](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@matthieu](https://discourse.julialang.org/u/matthieu)\
**Post date:** [January 29, 2019, 9:07pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/29 "2019-01-29T21:07:25Z")

</div>

The length of the SkipMissingArrays would be exactly the same as the underlying array. Think of it as a wrapper that basically applies the option `skipmissing = true` by default to `mean`, `skewness` etc.  
I’m not sure it is the answer to all the solutions, but that may be one way to implement automatic skipmissing though external packages.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [January 29, 2019, 9:31pm UTC](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146/30 "2019-01-29T21:31:39Z")

</div>

I had thought about that before, and I don’t think that can work. The `AbstractArray` interface assumes that the length and the number of indices are consistent with each other. I guess you could return indices including missing values, and only skip missing values during iteration, but many functions do `for i in eachindex(X); X[i]...` rather than `for x in X; x...`, in which case you will have to either return `missing` or throw an error for missing values.

[Previous page](https://discourse.julialang.org/t/how-does-statsbase-skewness-work/20146.md?page=1)