# Can Julia implement experiment animations similar to those in OpenAI Gym?

**URL:** <https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426>\
**Category:** Machine Learning\
**Tags:** question\
**Created:** [May 26, 2023, 6:08am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426 "2023-05-26T06:08:00Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [May 26, 2023, 6:08am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/1 "2023-05-26T06:08:00Z")

</div>

When doing reinforcement learning tasks, OpenAI Gym is commonly used for experimentation, for example, solving classic problems such as Cart Pole and Pendulum. While `ReinforcementLearning.jl` also has built-in environments for these problems, I would like to ask whether `ReinforcementLearning.jl` supports displaying animation effects like the one shown below, and how it can be implemented in Julia?

![314e251f95cad1c8f1959f1e723e6709c93d51d8](https://global.discourse-cdn.com/julialang/original/3X/e/6/e65467d80754f045945fd2b9cf7b5dbfd166e583.gif)

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [May 26, 2023, 6:12am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/2 "2023-05-26T06:12:27Z")

</div>

You can do animations with Makie  
[https://docs.makie.org/stable/documentation/animation/index.html#animations](https://docs.makie.org/stable/documentation/animation/index.html#animations)

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [May 26, 2023, 6:16am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/3 "2023-05-26T06:16:58Z")

</div>

Thank you, this is a good thing worth learning! However, what I’m more interested in is whether there is a visualization experimental environment in Julia for testing the effectiveness of reinforcement learning at present?

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [May 26, 2023, 7:08am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/4 "2023-05-26T07:08:35Z")

</div>

There is default plotting for a few environments, you can see them [here](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/blob/main/src/ReinforcementLearningEnvironments/src/plots.jl).

So you would probably have to call this yourself each step to get an animation, though I think that should be easy to do with a callback.

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [May 26, 2023, 7:31am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/5 "2023-05-26T07:31:05Z")

</div>

Thank you! I will study it. By the way, I would like to ask if there are any detailed explanations for beginners about the built-in environments in ReinforcementLearning.jl?

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [May 26, 2023, 7:38am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/6 "2023-05-26T07:38:01Z")

</div>

Not sure, all I could find was [this](https://juliareinforcementlearning.org/docs/How_to_write_a_customized_environment/) from the docs.

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [May 26, 2023, 7:39am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/7 "2023-05-26T07:39:42Z")

</div>

Me too, thanks!

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 26, 2023, 5:33pm UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/8 "2023-05-26T17:33:56Z")

</div>

Just a reminder that this might be very slow since the whole plot has to be regenerated for every step. The strength of Makie animations is that you can choose to only animate certain things, which is much cheaper computationally

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [May 26, 2023, 8:29pm UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/9 "2023-05-26T20:29:49Z")

</div>

That is probably true.

Though I wouldn’t recommend running this for every step of the training anyway, would generate too much data. And just running it for evaluation every now and then it might not matter too much about the speed.

Though we don’t really know what environment OP was interested in, and if they plan on creating their own animation for one. So maybe it could be nice to try in Makie to open up for the higher speed if it should be used in the background.

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [May 27, 2023, 8:59am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/10 "2023-05-27T08:59:36Z")

</div>

The pole will shorten when it falls…

![image](https://global.discourse-cdn.com/julialang/original/3X/6/6/6604c8d88adb03e1b16d1b4689bbcba81e7222f1.png)  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/1/d/1d69880bc9be8b2de1b696a0eb2c4af9e4949a05.png)  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/b/fb2df1b2fe8fd6f3b195257b979719000b198b73.png)

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [May 27, 2023, 11:41am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/11 "2023-05-27T11:41:35Z")

</div>

Hmm, unclear to me. Never really used the plotting, just remembered having seen it existed.

Looking at the code it is not obvious to me what the problem is, assuming theta=0 means up the plotting of the arm seems sensible to me, so can’t really say why it would behave as described.

Maybe easiest to just roll your own plotting since this seems to not really be well maintained things, no tutorial using them and so on.

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [May 27, 2023, 11:48am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/12 "2023-05-27T11:48:19Z")

</div>

Thank you very much! Your reminder is very helpful to me. 🤝

---

<div class="post-metadata">

**Author:** ![greatpet](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@greatpet](https://discourse.julialang.org/u/greatpet)\
**Post date:** [January 28, 2025, 11:39am UTC](https://discourse.julialang.org/t/can-julia-implement-experiment-animations-similar-to-those-in-openai-gym/99426/13 "2025-01-28T11:39:01Z")

</div>

Bumping this old thread which has been useful to me. I took the animation function mentioned here but used a different solver. Here’s my CartPole result:

> [@Reinforcement learning packages for CartPole example with Julia v1.11 or v1.10?](https://discourse.julialang.org/t/reinforcement-learning-packages-for-cartpole-example-with-julia-v1-11-or-v1-10/125261/3):
>
> I got it working. Here’s an animation of a trained agent balancing a pole on a cart, after training with 8000 steps of interactions. The arrow indicates if the agent is pushing the cart to the left or to the right. [anim] I used ReinforcementLearning.jl which provided the CartPole environment. However, most of the NN solvers in this package seem to be broken due to API changes in dependencies, therefore I used the solver in DeepQLearning.jl after converting the CartPole environment to its inte…