# Probabilistic programming with source transformations

**URL:** <https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047>\
**Category:** Statistics\
**Tags:** announcement\
**Created:** [December 28, 2017, 8:45pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047 "2017-12-28T20:45:54Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 31, 2017, 1:51am UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/21 "2017-12-31T01:51:18Z")

</div>

Cool, we’ll keep in touch then. No reason to rush. I hope to get a GSoC project devoted to Bayesian parameter estimation, in which case it would be nice to get setup with a few Julia tools and compare between them, but that would be months off (and requires finding a student 😄)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 31, 2017, 7:40am UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/22 "2017-12-31T07:40:30Z")

</div>

> [@cscherrer](#):
>
> some of the \mathbb{R}^n bijections will take some work.

I have a working version of the LKJ prior for correlation matrices on my computer that I will commit within a few days to ContinuousTransformations.jl, then we just need stick breaking and unit vectors, and have all the transformations Stan supports out of the box. If you need any of them quickly, I am happy to add them, just open an issue.

---

<div class="post-metadata">

**Author:** ![Liso](https://avatars.discourse-cdn.com/v4/letter/l/898d66/32.png) [@Liso](https://discourse.julialang.org/u/Liso)\
**Post date:** [December 31, 2017, 10:18am UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/23 "2017-12-31T10:18:10Z")

</div>

It seems to be breaking change. Is it possible to have it in 1.0? That issue is `speculative` without any milestone!

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [December 31, 2017, 3:56pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/24 "2017-12-31T15:56:41Z")

</div>

> [@Tamas\_Papp](#):
>
> I have a working version of the LKJ prior for correlation matrices on my computer that I will commit within a few days to ContinuousTransformations.jl, then we just need stick breaking and unit vectors, and have all the transformations Stan supports out of the box. If you need any of them quickly, I am happy to add them, just open an issue.

Nice! I think a natural way to connect this is that for the support of a distribution to determine the transformation. `Distributions.support` is currently only defined for univariate continuous distributions, but in principle it should be easy to extend.

Your `DynamicHMC` work seems to be set up specifically for `ReverseDiff`. From what I’ve seen, I think `XGrad` would be a little simpler and faster, and makes it pretty easy to specify optimized derivatives. For example, instead of examining the code for a given pdf, we know to call the cdf directly.

It will be great to get even a simple example running end-to-end:

- Find the supports
- Determine and apply the appropriate transformations
- Adjust for the log-Jacobian
- Compute the gradient
- Call NUTS

I’m sure I’ll be asking for help with a lot of this 🙂

> [@Liso](#):
>
> It seems to be breaking change. Is it possible to have it in 1.0? That issue is speculative without any milestone!

Sorry, I don’t understand what this is referring to. Did you maybe post in the wrong thread?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 31, 2017, 4:11pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/25 "2017-12-31T16:11:31Z")

</div>

> [@cscherrer](#):
>
> Your DynamicHMC work seems to be set up specifically for ReverseDiff.

It is set up to expect the log density and its gradient in a [`DiffResult`](https://github.com/JuliaDiff/DiffResults.jl) structure. I usually get that using ForwardDiff.jl, but ReverseDiff.jl is an option, too.

> [@cscherrer](#):
>
> It will be great to get even a simple example running end-to-end:

I plan to do a few of those that once `v0.7` lands, because it is really much nicer with named tuples. But if I have some time I will do a Julia notebook next week for a simple problem.

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [December 31, 2017, 4:20pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/26 "2017-12-31T16:20:23Z")

</div>

> [@Tamas\_Papp](#):
>
> It is set up to expect the log density and its gradient in a DiffResult structure. I usually get that using ForwardDiff.jl, but ReverseDiff.jl is an option, too.

Curious why you use `ForwardDiff` – just a legacy thing? Have you tried [`XGrad`](https://github.com/dfdx/XGrad.jl) or [`AutoGrad`](https://github.com/denizyuret/AutoGrad.jl)? I still don’t understand all of the autodiff tradeoffs, but for gradients reverse-mode is generally better. And I really prefer to be able to specify gradients manually if possible.

> [@Tamas\_Papp](#):
>
> I plan to do a few of those that once v0.7 lands, because it is really much nicer with named tuples. But if I have some time I will do a Julia notebook next week for a simple problem.

That would be really great, even something very simple with all of the pieces in place would be really helpful!

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 31, 2017, 4:23pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/27 "2017-12-31T16:23:12Z")

</div>

You are right, for gradients reverse mode is preferable; I mainly use ForwardDiff because it is very robust and fast enough for my purposes. I tried ReverseDiff a while ago and ran into problems, but I should look at it again because many of those issues were fixed in the meantime, I have not tried AutoGrad. XGrad is out for me because I don’t have closed form likelihoods (using indirect inference).

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [December 31, 2017, 4:36pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/28 "2017-12-31T16:36:01Z")

</div>

> [@Tamas\_Papp](#):
>
> XGrad is out for me because I don’t have closed form likelihoods (using indirect inference).

Interesting, this sounds a lot like more general “universal” probabilistic programming approaches, like what [Turing](http://turing.guru/) does. I don’t yet understand exact boundary of where `XGrad` will and won’t work, but for the Stan-like approach I think it should be a good fit. Other inference algorithms can just call something else 🙂

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [December 31, 2017, 6:09pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/29 "2017-12-31T18:09:32Z")

</div>

> [@Tamas\_Papp](#):
>
> XGrad is out for me because I don’t have closed form likelihoods (using indirect inference).

Can you provide an example of this? XGrad has its limitations, but they aren’t much different from those of ReverseDiff, so perhaps I’m missing some important use case.

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [January 2, 2018, 11:07pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/30 "2018-01-02T23:07:19Z")

</div>

Some more progress:  
[https://github.com/cscherrer/Soss.jl](https://github.com/cscherrer/Soss.jl)

@Tamas_Papp: I played with your `ContinuousTransformations` library a bit. I really like it, but I need the source code for a transformation and its Jacobian. Looks like maybe your stuff would need to be extended to get at this. For now I just hand-coded a couple of cases, and there’s no inverse transform stuff yet. Oh, and I also uploaded my updated NUTS code. But as you pointed out, this is from the original paper. It will be much better to get the updated version going.

@ChrisRackauckas: This notation is really similar to Turing’s, so it should be really easy to set up Turing as a back-end and vice versa. I would have worked with Turing directly, but the source code transformation approach seemed pretty different from what you guys are doing.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [January 3, 2018, 8:07am UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/31 "2018-01-03T08:07:53Z")

</div>

Just curious - how is the scope and goals of this package distinct from Turing?

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [January 3, 2018, 4:12pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/32 "2018-01-03T16:12:23Z")

</div>

> [@mkborregaard](#):
>
> Just curious - how is the scope and goals of this package distinct from Turing?

Goals are very similar. But the approach is very different, which I expect will lead to different strengths and ideal use cases.

Turing expresses a model as a function, in terms of an overloaded `~` operator. Changing inference methods means changing the way this is evaluated at runtime.

Soss focuses instead of source rewriting. For example, using a Stan-like approach, it can transform this model:

```julia
normalModel = quote
    μ ~ Normal(0,5)
    σ ~ Truncated(Cauchy(0,3), 0, Inf)
    for x in DATA
        x <~ Normal(μ,σ)
    end
end

```

into

```julia
:(function (θ, DATA)
        ℓ = 0.0
        μ = θ[1]
        ℓ += logpdf(Normal(0, 5), μ)
        σ = softplus(θ[2])
        ℓ += abs(σ - θ[2])
        ℓ += logpdf(Truncated(Cauchy(0, 3), 0, Inf), σ)
        for x = DATA
            ℓ += logpdf(Normal(μ, σ), x)
        end
        return ℓ
    end)

```

In sampling or optimization, this function becomes part of an inner loop, so performance is critical. It’s not obvious here, but parameters are always rewritten as a vector. There’s no unboxing overhead – instead we can just index into θ. In principle, we can add arbitrarily complex transformations along similar lines.

I guess my focus is on avoiding the “really neat, but too slow” critique common in probabilistic programming. I know the biggest overhead in Stan is the compile time. RStan is great, but uses one language (`R`) for data prep, another (Stan) for modeling, and a third (C++) for implementation under the hood. Results of a model are then analyzed back in R. Doing all of this in one language should help, and the language properties and tools available in Julia make it an obvious choice.

Oh, also… Stan gets some of its performance from the constraints it imposes on models (fixed-dimension continuous parameter space). This makes it a natural place to start. But in principle, Soss can be a lot more flexible. I’ve been thinking about this for a long time, but the implementation itself is less than a week old, so I’m still feeling out where in the design space it will land.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 3, 2018, 4:16pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/33 "2018-01-03T16:16:19Z")

</div>

> [@cscherrer](#):
>
> In sampling or optimization, this function becomes part of an inner loop, so performance is critical. It’s not obvious here, but parameters are always rewritten as a vector. There’s no unboxing overhead – instead we can just index into θ. In principle, we can add arbitrarily complex transformations along similar lines.

Probably should be static vectors then?

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [January 3, 2018, 4:19pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/34 "2018-01-03T16:19:00Z")

</div>

> [@ChrisRackauckas](#):
>
> Probably should be static vectors then?

Good point! Yes, definitely.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [January 3, 2018, 11:26pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/35 "2018-01-03T23:26:35Z")

</div>

Thanks, sounds very interesting!

---

<div class="post-metadata">

**Author:** ![nurban](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nurban/32/5278_2.png) [@nurban](https://discourse.julialang.org/u/nurban)\
**Post date:** [January 3, 2018, 11:42pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/36 "2018-01-03T23:42:02Z")

</div>

Is the limitation of using Stan only through a source transformation going to persist for the foreseeable future? That is, could Stan.jl eventually support calling out to arbitrary user-defined Julia functions that return gradient information via AD, or is the Stan engine not flexible enough to support that? I’d like to mix Stan with custom Julia functions, including ODE solvers; I’m not sure I trust Turing.jl or other backends for the statistical models I’m developing.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 3, 2018, 11:53pm UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/37 "2018-01-03T23:53:55Z")

</div>

> [@nurban](#):
>
> That is, could Stan.jl eventually support calling out to arbitrary user-defined Julia functions that return gradient information via AD, or is the Stan engine not flexible enough to support that?

I’m not sure it can, at least not easily. At least I don’t see FFI in the Stan manual, and if there is, calling into Julia would still be non-trivial.

> [@nurban](#):
>
> I’m not sure I trust Turing.jl or other backends for the statistical models I’m developing.

Not sure what trust has to do with it. But in DiffEqBayes.jl we’ll be developing the same interface over a bunch of different backends so by testing on the same problems we’ll have a more quantitative form of trust.

> **[GitHub - SciML/DiffEqBayes.jl: Extension functionality which uses Stan.jl,...](https://github.com/SciML/DiffEqBayes.jl)**
>
> Extension functionality which uses Stan.jl, DynamicHMC.jl, and Turing.jl to estimate the parameters to differential equations and perform Bayesian probabilistic scientific machine learning - GitHub...

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [January 4, 2018, 12:29am UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/38 "2018-01-04T00:29:54Z")

</div>

> [@nurban](#):
>
> Is the limitation of using Stan only through a source transformation going to persist for the foreseeable future? That is, could Stan.jl eventually support calling out to arbitrary user-defined Julia functions that return gradient information via AD, or is the Stan engine not flexible enough to support that?

I’m a little confused what you mean by this. Soss does source code transformations and will implement similar methods to Stan. But it does not call Stan.

My understanding is that Stan.jl and PyStan both call the command-line version of Stan. RStan is a little more integrated, since it uses a bridge to the underlying C++ code Stan is written in. But all three pass Stan a model in the form of a string.

So Stan.jl calls (command-line) Stan, which compiles to C++. I guess in principle one could have the C++ Stan codebase call back to Julia.

> [@nurban](#):
>
> I’m not sure I trust Turing.jl or other backends for the statistical models I’m developing.

I can understand putting more trust in Stan, just because it’s more mature and more widely-used. But things can still be very fiddly, depending on parameterizations etc. Oh, and mixture models are very awkward, and easy to mess up. And sometimes the log-Jacobian needs to be updated by hand.

I’m not saying Turing or Soss are (or will be) better. Parameterization for some models can just be tricky. Model checking tools can help do some sanity checks, so there’s some great potential there. And Soss can more easily express mixtures, since there’s a nice combinator for those in Julia. In principle Stan could add this, but it would take a huge amount of work for anyone who’s not a core developer. Compared to a single-language approach, Stan’s extensibility isn’t so great.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 4, 2018, 7:38am UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/39 "2018-01-04T07:38:19Z")

</div>

> [@nurban](#):
>
> could Stan.jl eventually support calling out to arbitrary user-defined Julia functions that return gradient information via AD, or is the Stan engine not flexible enough to support that?

Since Stan supports function definitions, one could write a Julia → Stan function rewriting engine for a limited subset of Julia. Personally, for reasons explained above, I don’t think this is the way to go, but you can experiment.

> [@nurban](#):
>
> I’m not sure I trust Turing.jl or other backends for the statistical models I’m developing.

If you are doing anything nontrivial, I assume you are already validating your inference with simulated data, so I am not sure what your concern is. Also, if you like a framework but you are concerned about test coverage, you could contribute tests.

I think that the Stan engines for inference are quite mature and robust, but the language itself can easily lead to error-prone code. Debugging is difficult (with `printf` most of the time), unit testing components of your models is out (I have not seen a feasible approach for this, and it is a practical concern for models with \> 100 LOC). Also, it is easy to make conceptual/arithmetic mistakes for manually coded Jacobians.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 4, 2018, 8:08am UTC](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047/40 "2018-01-04T08:08:36Z")

</div>

> [@Tamas\_Papp](#):
>
> Since Stan supports function definitions, one could write a Julia -\> Stan function rewriting engine for a limited subset of Julia. Personally, for reasons explained above, I don’t think this is the way to go, but you can experiment.

DiffEqBayes.jl does this. It works only for functions defined by the ParameterizedFunctions because we need the information in a DSL to know how to do the transformation. So yeah, limited rewriting engines work, but you do need a good enough DSL. I wouldn’t see this as necessary unless the DSL already exists though, since if it doesn’t exist then you might as well use Stan’s.

> [@Tamas\_Papp](#):
>
> If you are doing anything nontrivial, I assume you are already validating your inference with simulated data, so I am not sure what your concern is.

@nurban mentioned he was going to be using the ODE solvers, and this is probably the easiest thing to check because you just stick the parameters in and see if your line goes sufficiently through the data points. Then you can also check against the result given by maximum likelihood estimation and see if the peak is close. You should do this even with Stan. It’s not difficult to come up with simple problem where Stan will have problems that would be caught with these checks (along with the standards of checking that the chains converged and all of that).

[Previous page](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047.md?page=1)

[Next page](https://discourse.julialang.org/t/probabilistic-programming-with-source-transformations/8047.md?page=3)
