# Request for un'stdlibfication of Statistics

**URL:** <https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443>\
**Category:** Internals & Design\
**Tags:** statistics, community\
**Created:** [June 8, 2022, 3:35pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443 "2022-06-08T15:35:36Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 8, 2022, 4:52pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/21 "2022-06-08T16:52:47Z")

</div>

The multiplicity of `Stat_.jl` has been notably a recurring issue for new and more experienced Julians. There is an unpersuasive historical story to this, imo no longer of any relevance except for the stickiness it has exhibited.

It would be more becoming for Julia to have review and reshuffle the statistical package’s supportive membrane to magnify intra-functional simplicity and inter-operational effervescence (figuratively and factually). Re-looking and agreeing on some few helpful and unharmful moves may garner the active focus that does it.

Ping me for the third participant 🧑🏻‍💻 .

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 8, 2022, 4:55pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/22 "2022-06-08T16:55:55Z")

</div>

Such an elegant exposition @JeffreySarnoff . I wish that my English skills will become that good someday 🙂

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 8, 2022, 4:57pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/23 "2022-06-08T16:57:53Z")

</div>

Moving Statistics out of the standard libraries seems like a good idea to me overall. However, what about `mean`? Can we move `mean` back into Base? 😂

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [June 8, 2022, 4:57pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/24 "2022-06-08T16:57:53Z")

</div>

Regarding the `dims` argument, I have [a PR](https://github.com/nalimilan/FreqTables.jl/pull/54) to support `pairwise(f, eachcol(tbl))` on Tables in FreqTables.jl. AFAICT this kind of pattern would replace most uses of `dims` while being more explicit and working both for arrays and tables (arrays are useful, I don’t think we can require using tables).

But for this to work we need changes in Tables.jl so that `eachcol` returns an `AbstractColumns` object so that we can dispatch on it. Otherwise there’s no way to know whether an object is a table or not (which is a problem in many places).

Another difficulty is that there’s no standard package for arrays with names. FreqTables uses NamedArrays, but there are other options (AxisArrays, AxisKeys…) without any clear winner, so it’s hard to decide that StatsBase or another central package should depend on one of them. Yet as soon as you take a Tables.jl object as an input, you’ll want to preserve column names by returning a named array… I think the solution for now is to develop a specialized package (e.g. TableStatistics.jl) until it’s clear that we want to merge these features into Statistics/StatsBase.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 8, 2022, 5:00pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/25 "2022-06-08T17:00:12Z")

</div>

> [@nalimilan](#):
>
> But for this to work we need changes in Tables.jl so that `eachcol` returns an `AbstractColumns` object so that we can dispatch on it. Otherwise there’s no way to know whether an object is a table or not (which is a problem in many places).

This problem exists because you are trying to support arrays in parallel to tables.

> [@nalimilan](#):
>
> Another difficulty is that there’s no standard package for arrays with names.

Of course there is!

```julia
coltable = (a=[1,2,3], b=[4,5,6])

rowtable = [(a=1,b=4), (a=2,b=5), (a=3,b=6)]

```

all built-in types to Julia that implement the Tables.jl API.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 8, 2022, 5:03pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/26 "2022-06-08T17:03:01Z")

</div>

```julia
module Foo
  mean(x) = sum(x) / length(x)
end

```

will do it in most use cases. Unless you need extra options like weights etc, in which case you can certainly add a statistics-maintained package as a dependency for full-featured mean.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 8, 2022, 5:06pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/27 "2022-06-08T17:06:29Z")

</div>

Given your concern for new users and package discoverability, I would think you would prefer that new users be able to do this in a fresh Julia session:

```julia
julia> x = rand(4); mean(x)
0.6715977345843244

```

It’s bad enough that that doesn’t actually work in a vanilla Julia session (no installed packages) without first doing `using Statistics`.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [June 8, 2022, 5:06pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/28 "2022-06-08T17:06:42Z")

</div>

> [@juliohm](#):
>
> This problem exists because you are trying to support arrays in parallel to tables.

Not really, as one can dispatch on `AbstractArray` for these. The problem is rather that not everybody is happy with the idea of having essential stats packages depend on Tables.jl and on a package supporting named arrays.

> [@juliohm](#):
>
> Of course there is!
> 
> ```julia
> coltable = (a=[1,2,3], b=[4,5,6])
> 
> rowtable = [(a=1,b=4), (a=2,b=5), (a=3,b=6)]
> 
> ```
> 
> all built-in types to Julia.

And what kind of object would `pairwise(cor, eachcol(coltable))` (or `cor(eachcol(coltable))`) return? Returning a table wouldn’t be appropriate as the role of rows and columns in a correlation _matrix_ is symmetric. It really needs to be a matrix with row and column names.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 8, 2022, 5:07pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/29 "2022-06-08T17:07:22Z")

</div>

I would rather point beginners to a cohesive ecosystem for stats with tabular data. The function `mean` by itself doesn’t justify the current situation with Statistics.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 8, 2022, 5:14pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/30 "2022-06-08T17:14:37Z")

</div>

> [@nalimilan](#):
>
> The problem is rather that not everybody is happy with the idea of having essential stats packages depend on Tables.jl and on a package supporting named arrays.

Tables.jl is a super lightweight package with a very flexible API that is used for all kinds of things. I never encountered a single limitation with it doing all kinds of complicated geometric/statistical processing in Meshes.jl/GeoStats.jl for example. I really think that we, as a community, should stick to its API in favor of a more user-friendly experience for beginners. If someone really thinks that there is a real need for other concept that is not tabular data in the ecosystem, then they can work separately on a package for that. We shouldn’t compromise the experience of 99% of users because 1% feels unhappy with tabular representations in statistics.

> [@nalimilan](#):
>
> And what kind of object would `pairwise(cor, eachcol(coltable))` (or `cor(eachcol(coltable))`) return?

Yes, correlation/covariance matrices are not tables, they are matrices (i.e. arrays). I personally think that a good design should return an AxisArray (or any new custom type developed by us) so that we don’t loose track of the variable names ever during our statistical workflows.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [June 8, 2022, 5:24pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/31 "2022-06-08T17:24:43Z")

</div>

A tabular statistics package could have a correlation table function returning an `|x|y|cor|` table.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 8, 2022, 5:49pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/32 "2022-06-08T17:49:54Z")

</div>

Yes, it could be anything we want, but I doubt that use cases will favor this choice. We compute covariances matrices to perform matrix decompositions, sample Gaussian processes, etc. We never use them as tables. That is why it is important to think of use cases and stick to them throughout the ecosystem.

---

<div class="post-metadata">

**Author:** ![brenhinkeller](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brenhinkeller/32/36648_2.png) [@brenhinkeller](https://discourse.julialang.org/u/brenhinkeller)\
**Post date:** [June 8, 2022, 6:06pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/33 "2022-06-08T18:06:41Z")

</div>

> [@juliohm](#):
>
> I can but that is too noisy, confusing, just in order to get a name like `Statistics.mean` into scope for extension. I’d rather work in a cohesive ecosystem developed **by** statisticians than try to amend the current situation with the stdlib. Statistics is not the job of Julia core devs, they have other priorities. This stdlib just increases sysimages without major benefits.

I don’t think I understand the objection here. Consider the following:

- You write a package TableStatistics.jl under the Tables aegis that imports Statistics.mean
- You add new methods for `mean(t::Table)` etc. and export them
- Your users do `using TableStatistics` and now they can do statistics on Tables

That seems simple and elegant, no?

---

<div class="post-metadata">

**Author:** ![brenhinkeller](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brenhinkeller/32/36648_2.png) [@brenhinkeller](https://discourse.julialang.org/u/brenhinkeller)\
**Post date:** [June 8, 2022, 6:13pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/34 "2022-06-08T18:13:15Z")

</div>

Also, FWIW if we need more maintainers for the non-Tables plain Statistics ecosystem, I will happily volunteer.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 8, 2022, 6:18pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/35 "2022-06-08T18:18:14Z")

</div>

Tables.jl is a trait-based API without parent type. We can’t rely on multiple dispatch to disambiguate with methods designed for arrays. We need a complete redesign with tables in mind.

---

<div class="post-metadata">

**Author:** ![brenhinkeller](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brenhinkeller/32/36648_2.png) [@brenhinkeller](https://discourse.julialang.org/u/brenhinkeller)\
**Post date:** [June 8, 2022, 6:23pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/36 "2022-06-08T18:23:06Z")

</div>

Ah I see.

That brings up the whole traits/interface discussion, which we probably do eventually want language-level support for? In this particular case, it does make me wonder if it mightn’t have been simpler to have a package which just defines `abstract type Table end` and have everyone who wants to buy-in extend and subtype that? But that’s major digression.

---

<div class="post-metadata">

**Author:** ![mbaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbaz/32/17295_2.png) [@mbaz](https://discourse.julialang.org/u/mbaz)\
**Post date:** [June 8, 2022, 6:34pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/37 "2022-06-08T18:34:09Z")

</div>

> [@juliohm](#):
>
> We can’t rely on multiple dispatch to disambiguate with methods designed for arrays.

I don’t fully understand all the implications of this plan. Does it mean that one wouldn’t be able to do basic statistics (mean, variance, histograms, etc) on arrays? I hope this is not the case.

---

<div class="post-metadata">

**Author:** ![StevenWhitaker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevenwhitaker/32/9749_2.png) [@StevenWhitaker](https://discourse.julialang.org/u/StevenWhitaker)\
**Post date:** [June 8, 2022, 6:36pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/38 "2022-06-08T18:36:13Z")

</div>

> [@juliohm](#):
>
> Yes, we could maintain methods for arrays, but who is going to maintain those? Do we really need them when in practice most statistical tasks deal with tables (arrays + variable names)?

Pretty much the only functions from Statistics that I use are `mean` and `std`, and I _never_ call them with tables, so I would hope that at least those functions would maintain methods for arrays.

> [@juliohm](#):
>
> I would rather point beginners to a cohesive ecosystem for stats with tabular data.

I think this would be a bit off-putting for people (like me) who just want to compute the mean of a collection of values (i.e., an array).

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 8, 2022, 6:39pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/39 "2022-06-08T18:39:02Z")

</div>

> [@brenhinkeller](#):
>
> it mightn’t have been simpler to have a package which just defines `abstract type Table end` and have everyone who wants to buy-in extend and subtype that?

That doesn’t work for tables from Base, e.g. a vector of named tuples or a named tuple of vectors.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 8, 2022, 6:42pm UTC](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443/40 "2022-06-08T18:42:20Z")

</div>

It means that anyone doing serious stats that is much more than just computing mean std and histograms could leverage a high-level and clean api while a basic API for arrays could be easily crafted in parallel or a simple wrapper type like MLJ.table could be used to convert an array to have dummy names without performance issues.

[Previous page](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443.md?page=1)

[Next page](https://discourse.julialang.org/t/request-for-unstdlibfication-of-statistics/82443.md?page=3)