Language impediments to inference of real-time streaming models

Specific Domains Probabilistic programming
1

I’m debating whether to attempt a Julia solution to a particular “streaming” inference problem in which a global data model is constantly being updated with new data. I’m new to the language, but there seems to be language-related complications; Julia seems to favor statically-defined, rather than dynamically-generated, functions. I am aware Julia’s metaprogramming facilities, but would rather not delve into that if I can avoid it.

Here’s the problem. I want to perform inference on a global model (a parametric likelihood function) of real-time streaming data which is being modified in real time as follows. Incoming data is clustered; each cluster defines a parametric likelihood function of its constituent data; the global likelihood function is updated by multiplication by the likelihoods of the new data clusters. For example: Initial global model f_0(x|\theta); new-data likelihood g(y|\theta); updated likelihood is f(x,y|\theta)=f_0(x|\theta)\cdot g(y|\theta).

Questions:

  1. How to do inference on the current model? This seems problematic since the global likelihood function cannot be recompiled in the same Julia session. (True?) How then do I take code that performed inference against f_0 and point it at f(x,y)? What is the Julian approach? Use Revise? Rename the global model whenever I update with new data? My concern there is having potentially thousands of global models sitting around in memory indefinitely.
  2. Should I be worried about the latency of compiling the updated model? In the example above, f_0(x|\theta) may be very complex; g is relatively simple by comparison. When Julia compiles the new global model, f(x,y), does it efficiently reuse the previously compiled f_0, or is the latter recompiled along with g?
  3. Does anyone know of an existing Julia project doing something like this already?

Thanks!

2

I would use an approach with higher order functions, chaining together the likelihood incrementally.

3

You can define X as [x, y1, y2, y3] and define f(X) = f0(X[1]) * prod(g, X[2:end]). If you have more data, just push! it to X and call f again on X.

Edit: although if you are multiplying many such terms, I would work in terms of the log and add instead to avoid underflow.

Edit 2: to avoid re-computing terms, you can make f a callable struct with a cache field and save the results of f0(X[1]) and every computed value in the cache field in f. Alternatively, there is Memoization.jl to memoize the functions automatically.

1 Like
4

I did several projects related to BNP and dynamic compositional likelihoods. In my cases those compositions usually get quite complicated as they are also nested and, therefore, I always used an approach similar to what @mohamed82008 suggested for you in (2). But there are many approaches in Julia to do this and I in my experience Julia is one of the more suited language for this.

5

Thank you! Have you, by chance, used this approach with automatic-differentiation packages, in particular ReverseDiff? Do they play nicely together?

6

Thanks very much. Can you describe or point me to examples of the other approaches you alluded to?

7

ReverseDiff should be fine yes. If it is not, please open an issue with a minimal working example.