# How to design distributed (CPU/GPU) reinforcement learning methods?

**URL:** <https://discourse.julialang.org/t/how-to-design-distributed-cpu-gpu-reinforcement-learning-methods/20539>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [February 7, 2019, 10:17am UTC](https://discourse.julialang.org/t/how-to-design-distributed-cpu-gpu-reinforcement-learning-methods/20539 "2019-02-07T10:17:55Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [February 7, 2019, 10:17am UTC](https://discourse.julialang.org/t/how-to-design-distributed-cpu-gpu-reinforcement-learning-methods/20539/1 "2019-02-07T10:17:55Z")

</div>

Question to the experts on distributed computing:  
How should we design distributed reinforcement learning methods in julia?  
[Ray](https://github.com/ray-project/ray) has a nice approach with [different optimizers](https://github.com/ray-project/ray/tree/master/python/ray/rllib/optimizers) for methods that move around gradients or batches of experience, syncronoulsy or asyncronously.

As a super simple toy example of a method where agents collect experience on their own threads, send them to the learner and receive the updated policy I wrote the following:

```julia
using Distributed
@everywhere begin
    function actor(in, out)
        while true
            isdone, policy = take!(in)
            action = randn() + policy
            reward = 2 < action < 3
            put!(out, (myid(), action, reward))
            isdone && return
        end
    end
end

function learner(; policyinit = 0., η = 1e-2, T = 10^4)
    in = [RemoteChannel(()->Channel{Tuple{Bool,Float64}}(1)) for _ in 1:nprocs() - 1]
    out = [RemoteChannel(()->Channel{Tuple{Int,Float64,Bool}}(1)) for _ in 1:nprocs() - 1]
    for i in 1:nprocs() - 1
        remote_do(actor, i+1, in[i], out[i])
    end
    policy = policyinit
    for step in 1:T
        for i in 1:nprocs() - 1
            put!(in[i], (step == T, policy))
            id, action, reward = take!(out[i])
            policy += η * reward * (action - policy)
        end
    end
    policy
end

learner()

```

1. Does this look like a reasonable approach?
2. This works for CPUs. Can I follow a similar pattern with GPUs?
3. I have no clue about the performance of `RemoteChannel`. Do you think it will be possible to get competitive performance with this approach as compared e.g. to ray?