# Reinforcement Learning Custom Gym error

**URL:** <https://discourse.julialang.org/t/reinforcement-learning-custom-gym-error/87818>\
**Category:** Machine Learning\
**Created:** [September 26, 2022, 2:49pm UTC](https://discourse.julialang.org/t/reinforcement-learning-custom-gym-error/87818 "2022-09-26T14:49:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chunky](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@chunky](https://discourse.julialang.org/u/chunky)\
**Post date:** [September 26, 2022, 2:49pm UTC](https://discourse.julialang.org/t/reinforcement-learning-custom-gym-error/87818/1 "2022-09-26T14:49:15Z")

</div>

I’ve created a custom Gym environment using ReinforcementLearning.jl. I _think_ it adheres to the guidance on the documentation [“How to write a customised environment”] , but the trivial case to run my gym, using a random policy, throws an error.

The project is here: [GitHub - chunky/julia\_gym\_dll: Julia glue for a Gym written in C/C++](https://github.com/chunky/julia_gym_dll)

A notebook [committed, for my sins, with the error included], is here: [julia\_gym\_dll/Test\_DLLGym.ipynb at master · chunky/julia\_gym\_dll · GitHub](https://github.com/chunky/julia_gym_dll/blob/master/julia_src/Test_DLLGym.ipynb)  
The default everything there should be an inverted pendulum Gym, and it should be aligned with the pendulum example in ReinforcementLearning.jl

I’ll paste the error here:

```julia
env = DLLGymEnv();
run(RandomPolicy(), env, StopAfterStep(1000), TotalRewardPerEpisode())

MethodError: reducing over an empty collection is not allowed; consider supplying `init` to the reducer

Stacktrace:
  [1] reduce_empty(op::Base.MappingRF{Base.ExtremaMap{typeof(identity)}, typeof(Base._extrema_rf)}, #unused#::Type{Int64})
    @ Base ./reduce.jl:348
  [2] reduce_empty_iter
    @ ./reduce.jl:371 [inlined]
  [3] mapreduce_empty_iter(f::Function, op::Function, itr::Vector{Int64}, ItrEltype::Base.HasEltype)
    @ Base ./reduce.jl:367
  [4] _mapreduce
    @ ./reduce.jl:419 [inlined]
  [5] _mapreduce_dim
    @ ./reducedim.jl:365 [inlined]
  [6] #mapreduce#764
    @ ./reducedim.jl:357 [inlined]
  [7] mapreduce
    @ ./reducedim.jl:357 [inlined]
  [8] #_extrema#790
    @ ./reducedim.jl:999 [inlined]
  [9] _extrema
    @ ./reducedim.jl:999 [inlined]
 [10] #_extrema#789
    @ ./reducedim.jl:998 [inlined]
 [11] _extrema
    @ ./reducedim.jl:998 [inlined]
 [12] #extrema#787
    @ ./reducedim.jl:994 [inlined]
 [13] extrema
    @ ./reducedim.jl:994 [inlined]
 [14] extend_limits(vec::Vector{Int64}, limits::Tuple{Int64, Int64}, scale::typeof(identity))
    @ UnicodePlots ~/.julia/packages/UnicodePlots/Z7FG6/src/common.jl:371
 [15] UnicodePlots.Plot(x::Base.OneTo{Int64}, y::Vector{Float64}, z::Nothing, ::Type{UnicodePlots.BrailleCanvas}; title::String, xlabel::String, ylabel::String, zlabel::String, xscale::Symbol, yscale::Symbol, width::Nothing, height::Nothing, border::Symbol, compact::Bool, blend::Bool, xlim::Tuple{Int64, Int64}, ylim::Tuple{Int64, Int64}, margin::Int64, padding::Int64, labels::Bool, unicode_exponent::Bool, colorbar::Bool, colorbar_border::Symbol, colorbar_lim::Tuple{Int64, Int64}, colormap::Nothing, grid::Bool, xticks::Bool, yticks::Bool, min_width::Int64, min_height::Int64, projection::Nothing, axes3d::Bool, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ UnicodePlots ~/.julia/packages/UnicodePlots/Z7FG6/src/plot.jl:240
 [16] lineplot(x::Base.OneTo{Int64}, y::Vector{Float64}, z::Nothing; canvas::Type, color::Symbol, name::String, head_tail::Nothing, kw::Base.Pairs{Symbol, String, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:title, :xlabel, :ylabel), Tuple{String, String, String}}})
    @ UnicodePlots ~/.julia/packages/UnicodePlots/Z7FG6/src/interface/lineplot.jl:75
 [17] #lineplot#133
    @ ~/.julia/packages/UnicodePlots/Z7FG6/src/interface/lineplot.jl:79 [inlined]
 [18] (::TotalRewardPerEpisode)(#unused#::PostExperimentStage, agent::RandomPolicy{Nothing, Random._GLOBAL_RNG}, env::DLLGymEnv)
    @ ReinforcementLearningCore ~/.julia/packages/ReinforcementLearningCore/yeRLW/src/core/hooks.jl:160
 [19] _run(policy::RandomPolicy{Nothing, Random._GLOBAL_RNG}, env::DLLGymEnv, stop_condition::StopAfterStep{ProgressMeter.Progress}, hook::TotalRewardPerEpisode)
    @ ReinforcementLearningCore ~/.julia/packages/ReinforcementLearningCore/yeRLW/src/core/run.jl:48
 [20] run(policy::RandomPolicy{Nothing, Random._GLOBAL_RNG}, env::DLLGymEnv, stop_condition::StopAfterStep{ProgressMeter.Progress}, hook::TotalRewardPerEpisode)
    @ ReinforcementLearningCore ~/.julia/packages/ReinforcementLearningCore/yeRLW/src/core/run.jl:10
 [21] top-level scope
    @ In[10]:2
 [22] eval
    @ ./boot.jl:368 [inlined]
 [23] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1428

```

Any guidance would be appreciated  
Cheers  
Gary

---

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [September 26, 2022, 4:11pm UTC](https://discourse.julialang.org/t/reinforcement-learning-custom-gym-error/87818/2 "2022-09-26T16:11:37Z")

</div>

Just guessing based on the stacktrace: maybe the simulation stopped before a single episode ended and therefore an empty vector is passed to UnicodePlots, which produces this error message when you try e.g. `using UnicodePlots; Plot(Float64[], Float64[])`. Did you try to omit the plot? Maybe just adding a semicolon after the `run` function would be sufficient to suppress the plot and the error message?

---

<div class="post-metadata">

**Author:** ![chunky](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@chunky](https://discourse.julialang.org/u/chunky)\
**Post date:** [September 26, 2022, 10:35pm UTC](https://discourse.julialang.org/t/reinforcement-learning-custom-gym-error/87818/3 "2022-09-26T22:35:47Z")

</div>

You’re exactly right! The C gym’s return value of whether it’s done or not, is inverted compared to ReinforcementLearning.jl’s view.

Thanks!  
Gary