Some questions on Transducers.jl

General Usage
1

I’m recently interested in Transducers.jl.
However, I cannot understand some methods provided in the package.

For example, anybody can explain foldxl example in detail? I cannot figure out how the answer comes.

Also, is there any way to incorporate multiple inputs?
For example, I’d like to use a function f(x, y, z) = x+y+z like xs |> Scan(f, y0, z0) |> collect to compute functions recursively where xs = (x1, x2, ..., xn).

2

Transducers.Scan expects a 2-arg function as the first argument [1]. So, I’m not sure how you want to use f. Maybe it’s helpful to write down what you want to do in plain code (ok to be non-efficient or lengthy), so that people can know exactly what you want to compute.

  1. Transducers.jl APIs typically expect 2-arg (or sometimes 1-arg) function. ↩︎

2 Likes
3

Oh, you’re right. I should’ve attached an example.

I’m trying to write a code for a numerical simulator in FP style (for practice).

The following code is a simple Euler integration.
Since the differential equation is in general given as multi-input functions (like dyn(y, t)),
I need a multi-argument way to use Transducers.jl (or, ... functionality).
To reuse codes, multi-input support is necessary.

+) I’m interested in Transducers but I often feel it’s awkward and makes me hard to code fast. It would be delightful if you share your experience when using Transducers.jl and coding in FP style in aspects of 1) coding speed, 2) performance of codes, and 3) advantages.

julia> function euler(func)
           _euler(y, t, dt) = y + func(y, t) * dt
       end
euler (generic function with 1 method)

julia> dyn(y, t) = -y
dyn (generic function with 1 method)

julia> method = euler(dyn)
(::var"#_euler#2"{typeof(dyn)}) (generic function with 1 method)

julia> ts = 0:0.01:1
0.0:0.01:1.0

julia> ys = zeros(size(ts))
101-element Array{Float64,1}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 ⋮

julia> ys[1] = 2
2

julia> for i in 1:length(ts)-1
           ys[i+1] = method(ys[i], ts[i], ts[i+1]-ts[i])
       end

julia> ys
101-element Array{Float64,1}:
 2.0
 1.98
 1.9602
 1.940598
 1.9211920200000001
 1.9019800998
 1.882960298802
 1.86413069581398
 1.8454893888558404
 1.8270344949672819
 1.808764150017609
 1.790676508517433
 1.7727697434322587
 ⋮

EDIT: wrapping code makes it work as follows.

function integrator(func, y0, ts, method, args...; kwargs...)
    _ts = ts[1:end-1]
    Δts = diff(ts)
    propagate = method(func)
    _propagate(y, t_Δt) = propagate(y, t_Δt..., args...; kwargs...)
    ys = zip(_ts, Δts) |> Scan(_propagate, y0) |> collect
    return ys
end
4

Before anything, I think you’d want to check out DifferentialEquations.jl universe, if you haven’t.

A short answer for your question is that Tnrasducers.jl is not very beneficial for this type of computation, unless you need to do something very specific.

Having said that, it is certainly possible to use Transducers.jl for this

euler(dyn, y0, dt) = Scan(y0) do y, i
    y + dyn(y, i * dt) * dt
end
# e.g., 1:10 |> euler(dyn, 1.0, 0.01) |> collect

or, if you want the input to be the sequence of times:

euler2(dyn, y0, t0) = ScanEmit((y0, t0)) do (y, t_prev), t
    dt = t - t_prev
    y_next = y + dyn(y, t) * dt
    y_next, (y_next, t)
end
# e.g., 0.001:0.01:0.1 |> euler2(dyn, 1.0, 0.0) |> collect

(Side note: you can think of ScanEmit as a “discrete dynamical system executor”. So, if you can express your simulator as a discrete dynamical system, you can always use ScanEmit.)

However, I don’t think it is always a good idea to implement this kind of code with Trasudcers.jl, unless you have some specific technical requirements. For example:

  1. Fuse post-processing of the output of the system you are simulating; e.g., you are interested in a subset of the state variables or you want to filter out some outputs.
  2. Fuse event detection predicate functions in a very efficient manner.
  3. Simulating a system that is driven by input data (as in x(t) of dy/dt = f(y(t), x(t), t)) and the input data is stored in a complex data structure (e.g., nested array)

I’m probably the worst person to answer this question, since I designed the API for how I want to code.

But there are things very hard to express without transducers. For example, doing something like Transducers.Consecutive in parallel without transducers is very tricky (= very slow coding speed).

Transducers.jl APIs are fast sometimes and slow sometimes. I’ve been trying to reduce the slow cases but there are always possible improvements.

I think parallelism is the strongest selling point. Check out [ANN] Folds.jl: threaded, distributed, and GPU-based high-level data-parallel interface for Julia

3 Likes
5

Thanks a lot :slight_smile:

I’ve seen DifferentialEquations.jl but I sometimes felt it’s not suitable for my work including reinforcement learning and optimal control in continuous time with complex nested dynamical systems.

(I have no idea how to cite a certain part of your answer but) the examples you shared look nice.
It would be much easier to write codes with Transducers.jl if I follow your examples.

And I agree with that some functionality of Transducers.jl (especially ScanEmit) seems identical to discrete dynamical system and numerical simulation can be expressed as discrete dynamical system in general so I’ve been interested in numerical simulation using Transducers.jl.

Actually, I think that convenient post-processing and event detection are the most expected points when using Transducers.jl for numerical simulation. Thanks for clarifying it.

I’m also interested in Folds.jl.
I will practice and try to be familiar with Transducers universe.
Thank you again!

6

I should’ve mentioned that stateful transducers like the euler example I wrote is not parallelizable (in the sense you can’t do ... |> euler(dyn, y0, t) |> tcollect etc.). (Well, unless when the system is linear, but then you don’t need to simulate it.)

7

I agree with you.
It’s probably impossible in principle to apply parallelism to a single episode simulation for a causal dynamical system.
Instead, parallelism can be applied (very easily) to simulation of multiple episodes (for example, various initial conditions).

It’s actually already on progress to develop a package for transuder-based dynamical system simulator and I think that it provides flexible and convenient interface. For more information, see LazyFym.jl and a question about parallelism.

1 Like