Reworking Distributions.jl

I think the burden of proof is on your side. :wink: can you come up with a situation where you would like to dispatch/ work with the parameters of a distribution on the abstract type level without having a concrete type in mind? Especially in a way which cannot be handled using a function ˋparamsˋ?

2 Likes

Haha, ok. The most obvious is composition of Sampleables:

∘(::Sampleable{B,C}, ::Sampleable{A,B}) :: Sampleable{A,C}

[You could do this for Distributions as well, but implicitly this intergrates over the intermediate variable. Maybe we also need something that doesn’t? This could be similar to Gen’s concept of a ChoiceMap. @Marco_Cusumano-Towne or @alex-lew, any thoughts on this?]

Soss’s For combinator will simplify a lot:

For(::AbstractArray{P,N}, ::Distribution{P,X}) <: Distribution{AbstractArray{P,N}, X}

And it will be great to be able to more easily build Markov chains:

Unfold(::Distribution{P,X}, Distribution{X,X}) :: Distribution{P, Sequence{X}}

Sequence will be an iterator of some sort, but we don’t seem to have an AbstractIterator type (which is super weird, why don’t we have that?)

Note that this isn’t assuming we have “bulit-in” Distribution{X,X} types; the whole point of this is to be able to easily build things on the fly.

Some things to think about:

  • A given type will sometimes have a rand or logpdf, or neither, or both. The “logpdf but no rand” case isn’t covered by the original API, but it comes up, for example in energy-based models.
  • Will there ever be cases where we know a given type has a rand method, but we don’t realize there’s also a logpdf? Or vice-versa? Can this cause trouble?
  • Should we focusing so narrowly on distributions, rather than on measures more broadly? (#manifolds Slack with @sethaxen got me thinking about this)
  • If we represent measures, we should have a way to easily get to Lebesgue measure on the same space.
  • How can we bring mathematical laws in for use in tests?
  • If we went in that direction, could we still keep it easy to extend and add new distributions and combinators?
3 Likes

I think we had this discussion in the past, but I would prefer the Distributions.jl to be used for the well-known families of distributions people use as building blocks, and have a PDF, CDF, and IID random sampler available.

While technically all simulation codes or log pdfs define distributions, they lack either an IID random sampler or a pdf, so it is not very useful to use the same API for them in practice.

3 Likes

A few things about this are confusing to me:

  • As described above, this will not (at least at first) be Distributions.jl.
  • The current Distributions.jl includes Sampleable for the likelihood-free case
  • In Distributions.jl (and in life in general) CDF is typically only available for the one-dimensional case

This is currently true, at least in part because defining a Distribution or Sampleable currently takes a lot of work.

Overall, I think I’m missing what point you’re making here, could you give some more detail?

All I was suggesting is that I prefer Distributions.jl to be an implementation of the distributions which have some sort of a closed-form characterization (this is obviously subjective, but let’s say you need at least a pdf and IID sampling; CDF/quantiles are optional, having at least a mean is nice when known in closed form, other moments as available).

So, to me, the ideal Distributions.jl is just code implementing formulas and tables usually found in the appendix of some classical stats textbook or a paper. The “basic” stuff, which at the same time can be difficult to program because of all the finicky details.

Many other things are distributions, ie basically any finite measure can be used to define one, and outcome of a simulation also defines a distribution. I find it great that people want to experiment with describing these things in a single unified conceptual framework, and I follow this with interest, but I don’t think that Distributions.jl should be used to hammer out these ideas.

Of course, it it needs to be changed just a bit to support these projects, that’s great and it should be done. But I don’t think that Distributions.jl is the right place to think about distributions over arbitrary objects, DSLs to describe them, generalized sampling, etc.

I mostly agree with this. I’m not proposing a redesign, but a replacement. The “basic” stuff should of course be included. But the fact that we have a commonly used thing called a Distribution that’s so painfully non-extensible is something that needs to be improved.

We’re well past band-aids; the problem is fundamental to the design.

I see this a lot like the view of Julia from the outside. People say they’d rather have an expressive language than a fast one, without considering that they can have both.

I think it’s entirely possible to have an API that’s simpler than the existing one, and also extensible.

I don’t think anyone is questioning this, it’s just that the discussion would be more focused with an actual implementation of this API.

I agree, but it’s easier to lead with some discussion in case there are aspects of this I’m missing. I think my mental model of how this will work is converging, and I hope to get a proof of concept together over the next week or so

5 Likes

Looking forward to it!

2 Likes

Ari Katz (Are you on here Ari?) had a question on Slack #manifolds about GPU methods. This is another benefit of including the return type as a parameter - it’s easy to add a rand(SomeDistribution{P,X::CuArray}) method that specializes for GPU.

2 Likes

I’ll follow that with great interest. :+1:t2:

1 Like

Making some progress:
https://github.com/cscherrer/Measures.jl

6 Likes

What if I have a type that is Sampleable, but must participate in a completely different part of the type tree? I’m thinking about Particles <: Real or VAE <: AbstractGenerativeModel etc. These must be subtypes of their own type trees to work well, but can certainly be sampled from. The same with many probabilistic models etc., they often participate in their own type trees while acting as probability distributions at the same time. With this view point, it would be great if IsSampleable is a trait rather than an abstract type. Sorry for being late to the discussion, I didn’t realize this was potentially important to my application until now :confused:

A type-based workaround would be to implement wrapper types for everything that can also act like a probability distribution

This is great, then one could stick Particles in there, or any other form of prior distribution etc.

I would parameterize even harder to allow different types for mean and variance, consider e.g., that one wants to stick other distributions in there, such as Uniform(Normal(...), Gamma(...))

This suggestion would solve that problem, but may result in T=Any if one of the types is a Distribution whereas the other one is a VariationalAutoencoder, but maybe this is a pie-in-the-sky kind of application for which some dynamic dispatch would be acceptable?

That absolutely sounds worth considering

6 Likes

In total

abstract type Sampleable{X} end

should be more than enough and even Edit: that is too much

What if I have a type that is Sampleable, but must participate in a completely different part of the type tree?

You are absolutely right! I made these comments in situations where even more structure such as dimension, type of samples, type of probability, discreteness, uni- or multivariate was supposed to be baked into the abstract type. Having no abstract type, but a trait is very much preferable (or even no trait at all, just a set of functions with agreed behaviour, e.g. like we do it for iterators).

5 Likes

The current implementation is trait-based:

@trait IsMeasure{M,X} >: HasRand{M,X} where {X = eltype(M)} begin
    rand :: [M] => eltype(M)
end

I’ve actually started to wonder if we might need this anyway. Signed measures form a vector space over the reals, where addition is superposition. But if Distributions are also Measures, we’re getting into type piracy. So maybe we need a wrapper for working with external structures that happen to also be Measures?

I see the appeal of this, but going in this direction can cause a lot of problems. It’s important to distinguish a type from a measure on that type. On the other hand, some measures like a Dirichlet process really do take another measure as a parameter.

But you could build something like you describe as a mixture of uniforms over a product measure.

We definitely want to avoid dynamic dispatch if possible, ideally without preventing users from extending in this direction.

Right, I was looking at this last option a bit last night. Overall, we have

  • Hierarchy of abstract types, with wrappers for external interface
  • CanonicalTraits
  • Holy traits
  • Duck typing

I expect we’ll end up with some mixture of maybe two of these, still not entirely clear

3 Likes

Yeah, for the sampling part, a variation of the iterator protocol would be very natural,
querying new iterators from an iterator and new samples from a sampler is almost the same, the main differences is that there are two ingredients, the sampler and the random number generator. @rfourquet has some thoughts on this as well.

1 Like

Thanks for the ping @mschauer. I’ve followed this discussion not closely as I would like, in big part because I’m not familiar with the Distribution package nor do I need work with advanced distributions mathematically.

I wish to have time to understand better the needs for Distributions2, but one thing which would be great (IMHO) is that the design allows to put a basic Distribution type in Base and define there simple distributions like Normal or Uniform, which is also useful for the package for more advanced distributions. I tried to include such a type at one point, but one of the feedback was that it can’t really be done independently of the Distributions package.

adding constraints?

At the moment Distributions.jl has reached the point in its lifecycle that some people have issues with the interface/implementation, and are experimenting with various approaches and interface conventions (eg see this topic, and many others). It is unclear what the common ground will be, and whether there will be a consensus that leads to a refactoring of Distributions.jl, a fork, or an alternative package. There is no consensus about what a Distribution is, how to define the interface to be extensible (eg what if I want a distribution over arbitrary objects? is it beneficial to consider something I can only sample from using MCMC a distribution for the purposes of this type?), about implementation choices (type hierarchies? traits?).

Because of this, I disagree with putting anything to do with distributions this into Base, or even Random. I think it is better to decouple all development from Julia releases whenever possible, so that interfaces can evolve freely.

9 Likes