# Simplifying Distributions type hierarchy
**URL:** https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882
**Category:** Statistics
**Created:** [December 5, 2019, 12:55am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882 "2019-12-05T00:55:04Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@cscherrer](https://discourse.julialang.org/u/cscherrer)
#### Post date: [December 5, 2019, 12:55am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/1 "2019-12-05T00:55:04Z")
In probabilistic programming, we use a a lot of distributions. In some settings, it’s important for it to be very easy to define new distributions, and to reason about them.
The current type of a `Distribution` or `Sampleable` is
```julia
help?> Sampleable
search: Sampleable
Sampleable{F<:VariateForm,S<:ValueSupport}
Sampleable is any type able to produce random values. Parametrized by a
VariateForm defining the dimension of samples and a ValueSupport defining the
domain of possibly sampled values. Any Sampleable implements the Base.rand
method.
```
First we have to say whether a returned value is `Univariate`, `Multivariate`, or `Matrixvariate`. Then we have to say whether it’s `Discrete` or `Continuous`.
But all of this information is available from the type of the result. Why not just `Sampleable{T}`? Then…
- `sample(::Sampleable{T})` returns a value of type `T`
- `Sampleable{Int}` is clearly univariate discrete
- `Sampleable{Vector{Float}}` is multivariate continuous
In the current setup, say I want a distribution over trees. Then I have to define a new `Treevariate` type, and decide whether my trees will be continuous or discrete, or what that even means. `Sampleable{Tree{MoreParams}}` would do this just fine.
In practice, the heavy weight of implementing all of this often leads me to sidestep Distributions entirely. For example, in Soss I have an `EqualMix` combinator that gives a mixture model with equal-weight components. I define it as
```julia
struct EqualMix{T}
components::Vector{T}
end
```
and add some methods, and it works just fine. If I could tell it `T` should be a Distribution, that would be better. But the current setup doesn’t seem to me to map well to the domain.
Am I missing something? What’s the big benefit of the current approach?
---
### Author:  [@cpfiffer](https://discourse.julialang.org/u/cpfiffer)
#### Post date: [December 5, 2019, 2:44am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/2 "2019-12-05T02:44:18Z")
I was also very unclear on this. I can only imagine this allows some kind of fallback functionality for various distribution types.
---
### Author:  [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)
#### Post date: [December 5, 2019, 3:14am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/3 "2019-12-05T03:14:49Z")
Isn’t your Sampleable{T} in a trivial relationship with the current Sampleable{F\<:VariateForm,S\<:ValueSupport} if you have functions on T that specify (a) whether T is univariate or multivariate or (b) whether T is discrete or continuous? Looking through the Distributions.jl, there’s lots of methods defined on univariate types, which is what led to the original design IIRC.
I think your proposed hierarchy is more general and could be made to work. But is it better enough to go through a substantial refactoring of the code and potentially break everything out there?
---
### Author:  [@Ratingulate](https://discourse.julialang.org/u/Ratingulate)
#### Post date: [December 5, 2019, 3:24am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/4 "2019-12-05T03:24:14Z")
Can something be done with traits, which can then be defined on the old hierarchy so that existing code isn’t broken?
---
### Author:  [@cscherrer](https://discourse.julialang.org/u/cscherrer)
#### Post date: [December 5, 2019, 4:05am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/5 "2019-12-05T04:05:55Z")
> [@johnmyleswhite](#):
>
> But is it better enough to go through a substantial refactoring of the code and potentially break everything out there?
Here’s an example from Soss:
```julia
julia> m = @model x begin
α ~ Normal()
β ~ Normal()
σ ~ HalfNormal()
yhat = α .+ β .* x
n = length(x)
y ~ For(n) do j
Normal(yhat[j], σ)
end
end;
julia> rand(m(x=rand(3)))
(x = [0.4742281389391303, 0.8795517084789848, 0.48770790662969965], yhat = [0.7656025986035866, 0.69947236269799, 0.7634033181167845], n = 3, σ = 0.9023491416822307, β = -0.16315418316450547, α = 0.842974903245824, y = [-1.1024328623804904, 1.2723146474947935, -0.7741104398758925])
```
There should be a way to make a model built this way an instance of `Sampleable`. In a simpler approach the type would be really clear. How would you do this with the current definitions?
- It’s a huge amount of work for very little benefit. In the current setup it’s not worth the trouble
- It could be much easier
There are other irregularities, like how `check_args` is sometimes (but not always) available as a keyword argument. It’s a great library for what it was designed for, but PPLs need more. All of this just comes down to PPLs being much more demanding than classical statistical methods.
---
### Author:  [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)
#### Post date: [December 5, 2019, 7:09am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/6 "2019-12-05T07:09:23Z")
Perhaps, it’s worth writing `Distributions2.jl` to address the design concerns above and see if it provides enough benefits over the current design that making it the default is worth it.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [December 5, 2019, 9:19am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/7 "2019-12-05T09:19:05Z")
I think that the current approach comes from the early days of Julia when users were still exploring the language — most importantly, before traits were widely used. I have said this before in a previous discussion we had:
> [@Distributions.jl interop for probprog](https://discourse.julialang.org/t/distributions-jl-interop-for-probprog/26912/13):
>
> The first official release of Distributions.jl was in 2013, 6 years ago, before the actual release of Julia 0.3. IMO a lot of its design reflects that, eg it is full of ostensibly performance-enhancing checks and conversions which may not needed anymore, as the compiler got way smarter, the idiomatic Julia style for writing generic code has matured a lot since these days, using traits instead of elaborate type hierarchies, and duck typing where one can get away with it. This is natural …
I agree with @mohamed82008 that a rewrite would be best to demo a simpler setup. I would recommend much less inheritance and parameterization at the API level, and suggest traits instead (inheritance and type parameters could still be used to _implement_ a lot of functionality where they make sense). This could start as a PR, and depending on the reception it could either be merged or become Distributions2.
---
### Author:  [@mschauer](https://discourse.julialang.org/u/mschauer)
#### Post date: [December 5, 2019, 9:43am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/8 "2019-12-05T09:43:17Z")
The most important question (I think it is no question that the way distributions are parametrised is too restrictive) is whether the type of the samples `T` should be in the abstract type `Distribution{T}` or not `Distribution`. It does not have to be, one could either hook into the `eltype` trait or parallel infrastructure `sampletype` etc.
One question is whether we think of the distributions of `rand(Float64)` and `rand(Float32)` as same distribution with different precision or as different distributions? To experiment (and to solve some issues with the type hierarchy I was experiencing) I created
[https://github.com/mschauer/GaussianDistributions.jl](https://github.com/mschauer/GaussianDistributions.jl)
to show that a single `struct` `Gaussian` can represent univariate, multivariate normal distribution with various element types.
---
### Author:  [@mschauer](https://discourse.julialang.org/u/mschauer)
#### Post date: [December 5, 2019, 10:37am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/9 "2019-12-05T10:37:16Z")
Just a minor thing, showing the trickiness: the samples of `rand(Bool) + 1/2` are Float64 valued and have a discrete distribution.
---
### Author:  [@mbesancon](https://discourse.julialang.org/u/mbesancon)
#### Post date: [December 5, 2019, 10:57am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/10 "2019-12-05T10:57:48Z")
Like @mschauer mentioned, some edge cases are very tricky, we just hit some in some PRs.
The current PR tackling the type system issue is [https://github.com/JuliaStats/Distributions.jl/pull/951](https://github.com/JuliaStats/Distributions.jl/pull/951), just fixing some things on ValueSupport.
The trait discussion already happened a few months ago in another discourse thread, and it is more complex than it first appears. An idea I would submit is finish to #951 and a few things, release a last minor 0.22 (an equivalent of julia 0.7), and then release a 1.0. We can then experiment with a trait-based system for 2.0
---
### Author:  [@cscherrer](https://discourse.julialang.org/u/cscherrer)
#### Post date: [December 5, 2019, 7:10pm UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/11 "2019-12-05T19:10:54Z")
> [@Tamas\_Papp](#):
>
> I would recommend much less inheritance and parameterization at the API level, and suggest traits instead
I don’t understand the design space of traits well enough, can we be confident that they’ll solve some problems and create others? What’s the right way to think about the tradeoffs, and arrive at the right combination of parameterized types and traits?
Parameterized types can get us a lot further than where we are now if types were used in a more generic way. Currently we have 3 choices for `VariateForm` and two for `ValueSupport`, so there are really only 6 types of Distributions, and extending this is very slow. Parameterizing by the return type would get us a lot of mileage.
It could be helpful to set out some goals. Mine would include
- The return type should be easy to determine statically from the type (no `eltype` required)
- Extensibility should be easy and very lightweight (distributions over trees, named tuples, etc)
- “Combinators”, aka “higher-order” distributions should be easy. Here I’m thinking things like `MixtureModel` in Distributions, or `iid` or `For` in Soss.
- There should be a very flexible and extensible way of representing the support of a distribution, and this should represent the exact support, or indicate otherwise
- There should be a way to represent different parameterizations of a distribution easily and with no performance penalty
- Argument checking should be optional, with a uniform interface across distributions
> [@mschauer](#):
>
> Just a minor thing, showing the trickiness: the samples of `rand(Bool) + 1/2` are Float64 valued and have a discrete distribution.
Is the trickiness that one might want to build a distribution to represent values like this? It’s pretty common to work with distributions that are functions of or inverse functions of some known distribution. In this case what you describes has a bijection to `Bernoulli(0.5)`. Is it hard to create a type to represent this?
> [@mbesancon](#):
>
> The trait discussion already happened a few months ago in another discourse thread, and it is more complex than it first appears.
Is there evidence that traits are the right way to go, or what’s the “right way” to use them for this use case?
---
### Author:  [@sethaxen](https://discourse.julialang.org/u/sethaxen)
#### Post date: [December 6, 2019, 6:20am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/12 "2019-12-06T06:20:58Z")
> [@cscherrer](#):
>
> Parameterized types can get us a lot further than where we are now if types were used in a more generic way. Currently we have 3 choices for `VariateForm` and two for `ValueSupport` , so there are really only 6 types of Distributions, and extending this is very slow. Parameterizing by the return type would get us a lot of mileage.
We encountered the problems discussed here in [Manifolds.jl](https://github.com/JuliaNLSolvers/Manifolds.jl), and the way we’re handling it right now is with our own `VariateForm` and `ValueSupport` (e.g. see [https://github.com/JuliaNLSolvers/Manifolds.jl/blob/master/src/DistributionsBase.jl](https://github.com/JuliaNLSolvers/Manifolds.jl/blob/master/src/DistributionsBase.jl)). The support is typed on the manifold for dispatch. In some cases, we have a type parameter for the variables in the distribution similar to what you’re proposing (e.g. [https://github.com/JuliaNLSolvers/Manifolds.jl/blob/master/src/ProjectedDistribution.jl#L9-L13](https://github.com/JuliaNLSolvers/Manifolds.jl/blob/master/src/ProjectedDistribution.jl#L9-L13)).
In practice, we combine manifolds into a product manifold and their distributions into a [`ProductPointDistribution`](https://github.com/JuliaNLSolvers/Manifolds.jl/blob/master/src/ProductManifold.jl#L232-L267). At some point when we have support for maps, we’d like to have pushforward distributions. The result is that if we define a manifold and a type for a point on that manifold, we can also define a distribution for that point. I doubt they can just be used in any PPL without some tweaking though.
I’m interested in a traits-based approach and would love it if it would make defining these kinds of distributions more straightforward, though I don’t know what that would look like.
Edit: @mateuszbaran may have ideas; he implemented most of the distributions code in Manifolds.jl
---
### Author:  [@mateuszbaran](https://discourse.julialang.org/u/mateuszbaran)
#### Post date: [December 6, 2019, 11:01am UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/13 "2019-12-06T11:01:58Z")
Type hierarchy is `Distributions.jl` isn’t particularly well suited for manifold-valued data. For example `VariateForm` doesn’t really fit here. Representations of a single point or vector are very diverse in `Manifolds.jl` and I’ve basically opted out of `VariateForm` stuff by defining `MPointvariate` and `FVectorvariate`.
I like lightweight and trait-based solutions. The abstract type `Manifold` in `Manifolds.jl` doesn’t have any type parameters and it doesn’t cause any issues. What would be the benefits of enforcing some type parameters like `ValueSupport`? Anyone can put the return type in type signatures of derived types like I did in, for example, `ProjectedFVectorDistribution` ( @sethaxen described it quite well). Another point is that for example continuous distributions should generally put return type in the type signature.
---
### Author:  [@juliohm](https://discourse.julialang.org/u/juliohm)
#### Post date: [December 6, 2019, 1:41pm UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/14 "2019-12-06T13:41:40Z")
Can I just add to the discussion that the name Distributions2.jl feels strange? If you guys decide on a better design, can we just migrate it to the name Distributions.jl and make it a major release instead?
I dislike this convention that numbers are appended to package names, it doesn’t feel elegant.
---
### Author:  [@cscherrer](https://discourse.julialang.org/u/cscherrer)
#### Post date: [December 6, 2019, 3:21pm UTC](https://discourse.julialang.org/t/simplifying-distributions-type-hierarchy/31882/15 "2019-12-06T15:21:56Z")
> [@juliohm](#):
>
> If you guys decide on a better design, can we just migrate it to the name Distributions.jl and make it a major release instead?
I think everyone would prefer this, the challenge is just how to get there.
I guess it’s a common problem in software. Early-stage projects can turn on a dime, but slow down as they mature and other projects depend on them. Eventually, a new use case exposes blind spots in the interface. But now change is hard, and there are still open PRs with significant amounts of work put into them that further entrench the existing approach.
So, what’s the right way out of this? Development is most efficient when everyone contributing shares a common vision of the end result. Otherwise the PRs drag on and on, with more time spent negotiating and debating the merits than developing or actually using an implementation.
This was my hope in setting out some goals. If we can agree on these, we can start to move toward how to actually get there.In the other extreme, if PPL needs are different than general statistical use, it would probably have to be a different library altogether, perhaps calling to Distributions.jl for sampling, logpdf, etc, similar to the current Gen approach.