# How to perform an Off-Policy reinforcement learning using \`ReinforcementLearning.jl\`?

**URL:** <https://discourse.julialang.org/t/how-to-perform-an-off-policy-reinforcement-learning-using-reinforcementlearning-jl/99940>\
**Category:** Machine Learning\
**Tags:** question, package, machine-learning\
**Created:** [June 6, 2023, 1:46pm UTC](https://discourse.julialang.org/t/how-to-perform-an-off-policy-reinforcement-learning-using-reinforcementlearning-jl/99940 "2023-06-06T13:46:45Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [June 6, 2023, 1:46pm UTC](https://discourse.julialang.org/t/how-to-perform-an-off-policy-reinforcement-learning-using-reinforcementlearning-jl/99940/1 "2023-06-06T13:46:45Z")

</div>

I am a beginner of both Julia and RL. Now I’m trying to training a Q-based tabular policy with off-policy strategy using `ReinforcementLearning.jl`. Let `behavior_policy` be a `RandomPolicy` (for example) to collect experiences, I want the `target_policy` as a `QBasedPolicy` to be trained. I wrote the following code:

```julia
target_policy = QBasedPolicy(
    learner = MonteCarloLearner(;
        approximator=TabularQApproximator(
            ;n_state = length(state_space(wrapped_env)),
            n_action = length(action_space(wrapped_env)),
            opt = InvDecay(1)
        )
    ),
    explorer = EpsilonGreedyExplorer(0)
)

behavior_policy = RandomPolicy(action_space(wrapped_env))

p = OffPolicy(target_policy, behavior_policy)

agent = Agent(
    policy = p,
    trajectory = VectorSARTTrajectory()
)
hook = TotalRewardPerEpisode()
run(
    agent, 
    wrapped_env, 
    StopAfterEpisode(100), 
    hook
)

```

But it didn’t work as the `p.π_target.learner.approximator.table` still being all zero.

Is there anyone who can tell me where the problem lies? Do I take wrong understandings on off-policy reinforcement learning, or not properly use the `OffPolicy` method?

Thanks!