# Understanding MultiAgent in ReinforcementLearning.jl

**URL:** <https://discourse.julialang.org/t/understanding-multiagent-in-reinforcementlearning-jl/65269>\
**Category:** Machine Learning\
**Created:** [July 25, 2021, 4:06pm UTC](https://discourse.julialang.org/t/understanding-multiagent-in-reinforcementlearning-jl/65269 "2021-07-25T16:06:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![thae](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thae/32/27500_2.png) [@thae](https://discourse.julialang.org/u/thae)\
**Post date:** [July 25, 2021, 4:06pm UTC](https://discourse.julialang.org/t/understanding-multiagent-in-reinforcementlearning-jl/65269/1 "2021-07-25T16:06:11Z")

</div>

Hello, let it be known that I know little about Julia and even less about Reinforcement Learning.

I’ve written an environment so trivial it is meaningless just to understand how everything fits together. It’s a four-player game for players named 1, 2, 3 4. Each episode starts at player 1 and goes through 4. Each action simply updates env.last\_action to the last player. The reward is whatever is in env.last\_player. is\_terminated evaluates to true after the fourth player has played. Notice it’s a TERMINAL\_REWARD game, and it’s a meaningless game so no need to understand the logic. Both state and action exist but are worthless Int64s. The full code is at the bottom of this message.

I then throw four agents at the game.

So, questions:

1. While running this for the first time I got a MethodError like below after all four players played. I’m guessing that when the is\_terminated. I’m guessing that when is\_terminated happens, the approximator is called with (Int, NoOp) parameters instead of (Int, Int) - probably because the future state does not exist (and is a “NoOp”) ? Is this how it’s supposed to work or am I already wrong at this point?

Notice that to make it work I provided a (::Int64, ::NoOp) version of this method that does just replaces the NoOp with a different Int64, assuming that it needs a Int64 that maps to “end of game” or something like that.

```julia
ERROR: LoadError: MethodError: no method matching (::TabularQApproximator{Matrix{Float64}, Flux.Optimise.InvDecay})(::Int64, ::NoOp)
Closest candidates are:
  (::TabularQApproximator{T, O} where {T<:(AbstractArray{Float64, N} where N), O})(::Int64) at /home/thae/julia/packages/ReinforcementLearningCore/B6qPK/src/policies/q_based_policies/learners/approximators/tabular_approximator.jl:32
  (::TabularQApproximator{T, O} where {T<:(AbstractArray{Float64, N} where N), O})(::Int64, ::Int64) at /home/thae/julia/packages/ReinforcementLearningCore/B6qPK/src/policies/q_based_policies/learners/approximators/tabular_approximator.jl:33

```

1. After adding the I-assume-it’s-a-hack to fix the issue above (reproduced in the code below), I run the code. Notice that there are some printlns. Here’s how they run:

```julia
Action for player 1
reward(2)
reward(2)
reward(2)
reward(2)
Action for player 2
reward(3)
reward(3)
reward(3)
reward(3)
Action for player 3
reward(4)
reward(4)
reward(4)
reward(4)
Action for player 4
reward(1)
reward(1)
reward(1)
reward(1)

```

I assume this means that each of the agents (of which there are four) is polling for the rewards of the given player after each action. However:

- Why does it poll for rewards after each action if I am using TERMINAL\_REWARD? Shouldn’t it just peek into the rewards once at the end of the episode?

- The number there is the player whose reward is being polled. Notice that after player 1’s action the reward that is being polled is player 2’s. Why is this so? I’m guessing the problem is that at the end of the action I am moving the current player to the next one, so reward() gets called with the future player. How am I supposed to handle incrementing the player correctly?

- In this code there’s no meaningful reward logic but I want to assign a separate reward for each player at the end of the episode - I’d expect to see calls to reward() for each of the four players after the last step, not only for a single player. Do I misunderstand how TERMINAL\_REWARD works?

Thank you!

```julia
using ReinforcementLearning

mutable struct FooEnv <: AbstractEnv
  last_action::Int
  current_player::Int
end

RLBase.reset!(env::FooEnv) = FooEnv(0, 1)

RLBase.players(env::FooEnv) = collect(1:4)
RLBase.current_player(env::FooEnv) = env.current_player

RLBase.action_space(env::FooEnv, p) = [1,2,3]

RLBase.state_space(env::FooEnv) = Base.OneTo(2)
RLBase.state(env::FooEnv) = 1
(app::TabularQApproximator)(s::Int, a::NoOp) = app.table[3, s]

function (env::FooEnv)(action::Int, player)
  println("Action for player $player")
  env.last_action = player
  env.current_player += 1
  if env.current_player > 4
    env.current_player = 1
  end
end

RLBase.is_terminated(env::FooEnv) = env.last_action == 4

function RLBase.reward(env::FooEnv, player)
  println("reward($player)")
  env.last_action
end

RLBase.NumAgentStyle(::FooEnv) = MultiAgent(4)
RLBase.DynamicStyle(::FooEnv) = SEQUENTIAL
RLBase.StateStyle(::FooEnv) = Observation{Int}()
RLBase.RewardStyle(::FooEnv) = TERMINAL_REWARD
RLBase.UtilityStyle(::FooEnv) = GENERAL_SUM
RLBase.ChanceStyle(::FooEnv) = STOCHASTIC

env = FooEnv(0, 1)

#RLBase.test_runnable!(env)

function build_agent()
  approximator = TabularQApproximator(
                         ;n_state = length(state_space(env)),
                         n_action = length(action_space(env)),
                     )
  policy = QBasedPolicy(
             learner = MonteCarloLearner(;
                     approximator=approximator,
                     kind=EVERY_VISIT,
                 ),
             explorer = EpsilonGreedyExplorer(0.01, warmup_steps=10000, step=1)
         )

  agent = Agent(
    policy = policy,
    trajectory = VectorSARTTrajectory(; state=Int, action=Union{Int64,NoOp}, reward=Int, terminal=Bool)
  )
  agent
end

multiagent = MultiAgentManager(
    NamedPolicy(1 => build_agent()),
    NamedPolicy(2 => build_agent()),
    NamedPolicy(3 => build_agent()),
    NamedPolicy(4 => build_agent()),
)

run(multiagent, env, StopAfterEpisode(1))

```

---

<div class="post-metadata">

**Author:** ![findmyway](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/findmyway/32/4946_2.png) [@findmyway](https://discourse.julialang.org/u/findmyway)\
**Post date:** [July 25, 2021, 5:51pm UTC](https://discourse.julialang.org/t/understanding-multiagent-in-reinforcementlearning-jl/65269/2 "2021-07-25T17:51:53Z")

</div>

Hello @thae ,

All are good questions. The sad truth is, MARL related algorithms are not well explored in RL.jl. I’ll explain `MultiAgentManager` a little bit more first.

The most naive way to solve MARL problems is to transform them into single agent RL problems and then we can reuse many existing solutions (though doing so is problematic in many cases). That’s the basic idea of `MultiAgentManager` in RL.jl. And it has several assumptions:

- All the players play alternatively. Only one player is about to take an action at each step.
- After each step, all the players can observe the env independently, including state and reward.
- For players who are not the current player, an action of `NO_OP` is assumed to have been taken.
- The typical trajectory in a two-player environment is like this:

```julia
S1_p1, A1_p1, R1_p1, T1, S2_p1, NO_OP, r2_p1, ..., St_p1, At_p1, Rt_p1, Tt
S1_p2, NO_OP, R1_p2, T1, S2_p2, A2_p2, r2_p2, ..., St_p2, At_p1, Rt_p1, Tt

```

```julia
S -> State
A -> Action
R -> Reward
T -> Terminal
t -> time step
p1 -> player 1
p2 -> player 2

```

If any of the above assumptions doesn’t meet what you expected, then you’d better to write your own multi agent manager.

Now let me answer your question one by one.

> [@thae](#):
>
> 1. While running this for the first time I got a MethodError like below after all four players played. I’m guessing that when the is\_terminated. I’m guessing that when is\_terminated happens, the approximator is called with (Int, NoOp) parameters instead of (Int, Int) - probably because the future state does not exist (and is a “NoOp”) ? Is this how it’s supposed to work or am I already wrong at this point?

As I illustrated above, it is the ACTION taken by the the non-active players (in the view of `MultiAgentManager`). So you can simply set the element type of actions in the trajectory to be `Union{Int, NoOp}`. (You did it right!)

> [@thae](#):
>
> Why does it poll for rewards after each action if I am using TERMINAL\_REWARD? Shouldn’t it just peek into the rewards once at the end of the episode?

Yes, it assumes the env is of `StepReward` style. And do not treat `TerminalReward` style env especially. (We should definitely support it!)

> [@thae](#):
>
> The number there is the player whose reward is being polled. Notice that after player 1’s action the reward that is being polled is player 2’s. Why is this so? I’m guessing the problem is that at the end of the action I am moving the current player to the next one, so reward() gets called with the future player. How am I supposed to handle incrementing the player correctly?

You should call `reward(env, player)` to fetch/implement the reward of specific player. Otherwise, by default `reward(env)` only returns the reward of the current player.

> [@thae](#):
>
> In this code there’s no meaningful reward logic but I want to assign a separate reward for each player at the end of the episode - I’d expect to see calls to reward() for each of the four players after the last step, not only for a single player. Do I misunderstand how TERMINAL\_REWARD works?

This has nothing to do with `TERMINAL_REWARD` style environments. I think my above answer already answered this question?

> <https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/blob/63580d3e09975e90fbeabe971e3b93c44ffb2e4d/src/ReinforcementLearningBase/src/interface.jl#L532>

If this is quite counterintuitive, please file an issue 😀 at [Issues · JuliaReinforcementLearning/ReinforcementLearning.jl · GitHub](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues)

---

<div class="post-metadata">

**Author:** ![thae](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thae/32/27500_2.png) [@thae](https://discourse.julialang.org/u/thae)\
**Post date:** [July 27, 2021, 2:04pm UTC](https://discourse.julialang.org/t/understanding-multiagent-in-reinforcementlearning-jl/65269/3 "2021-07-27T14:04:44Z")

</div>

Thanks a lot! That answers most or all of my doubts. The typical two-player trajectory example was particularly useful.