# Want to learn Reinforcement Learning from scratch for mini-project

**URL:** <https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836>\
**Category:** Machine Learning\
**Tags:** question\
**Created:** [August 16, 2022, 4:19pm UTC](https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836 "2022-08-16T16:19:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lighfe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lighfe/32/38831_2.png) [@Lighfe](https://discourse.julialang.org/u/Lighfe)\
**Post date:** [August 16, 2022, 4:19pm UTC](https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836/1 "2022-08-16T16:19:55Z")

</div>

Hello everyone,

I’m quite new to Julia but have some basic knowledge of other programming languages.

As a project for myself I want to train an RL algorithm for a [boardgame](https://boardgamegeek.com/boardgame/244521/quacks-quedlinburg). It’s a push-your-luck type of game, where you need to asses risk/rewards and therefore I think there are 100% dominant winning strategies which can be learned.

On the same time this seems like a nice opportunity to get more familiar with Julia.

I have theoretical background knowledge on machine learning but not RL specific. So I basically want to teach myself Reinforcement Learning from scratch as well as the Julia specifics.  
Anybody has advice on how to approach this? I got quite some time on my hand during semester break. I just started reading “Reinforcement Learning: An Introduction” from Sutton & Barto, but I wonder how other people would approach this.

Thanks for any input/experiences 🙂

---

<div class="post-metadata">

**Author:** ![digital\_carver](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/digital_carver/32/33818_2.png) [@digital\_carver](https://discourse.julialang.org/u/digital_carver)\
**Post date:** [August 16, 2022, 6:40pm UTC](https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836/2 "2022-08-16T18:40:53Z")

</div>

I don’t have experience implementing RL in Julia, but I’ll just share a few links I’ve come across:

- [Getting Started With Reinforcement Learning in Julia](https://www.talkjulia.com/8) episode on the Talk Julia podcast.  
The page also has a few useful Episode links at the bottom of the page, especially the [An Introduction to ReinforcementLearning.jl: Design, Implementations and Thoughts](https://juliareinforcementlearning.org/blog/an_introduction_to_reinforcement_learning_jl_design_implementations_thoughts/) blog post.

- Two videos on [julia for talented amateurs](https://www.youtube.com/c/juliafortalentedamateurs) on YouTube. This channel is usually light and easy to follow, so is a good way to take first steps into something. The videos are [Markov Decision Process with POMDPs.jl](https://www.youtube.com/watch?v=GXnCm_6uyeM&list=PLhQ2JMBcfAsi76O13sJzk4LXA_mu5sd9E&index=12) and [SARSA and Q-learning Algorithms with POMDPs.jl](https://www.youtube.com/watch?v=ITzRizRpHzI&list=PLhQ2JMBcfAsi76O13sJzk4LXA_mu5sd9E&index=13).

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [August 16, 2022, 6:43pm UTC](https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836/3 "2022-08-16T18:43:43Z")

</div>

There is a [repo](https://github.com/JuliaReinforcementLearning/ReinforcementLearningAnIntroduction.jl) implementing solutions for all problems in “Reinforcement Learning: An Introduction” using the library [ReinforcementLearning.jl](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl), so that could maybe be a good start.

There is also the library [POMDPs.jl](https://github.com/JuliaPOMDP/POMDPs.jl) which also seem to cover many similar topics.

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [August 16, 2022, 7:11pm UTC](https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836/4 "2022-08-16T19:11:33Z")

</div>

Hi Lighfe,

As for books there are:

- “Algorithms for Decision Making”; Mykel J. Kochenderfer, Tim A. Wheeler, Kyle H. Wray; The MIT Press 2022: [https://mitpress.mit.edu/9780262047012/](https://mitpress.mit.edu/9780262047012/) AND [https://algorithmsbook.com/](https://algorithmsbook.com/)

- “Distributional Reinforcement Learning”; Marc G. Bellemare and Will Dabney and Mark Rowland; The MIT Press 2023 (draft under submission): [https://mitpress.mit.edu/9780262048019/distributional-reinforcement-learning/](https://mitpress.mit.edu/9780262048019/distributional-reinforcement-learning/) AND [https://www.distributional-rl.org/](https://www.distributional-rl.org/)

There are interesting videos:

- ReinforcementLearning.jl - “A Practical Introduction to ReinforcementLearning.jl” by Jun Tian: [ReinforcementLearning jl | Jun Tian | Julia User Group Munich - Share&Code - YouTube](https://www.youtube.com/watch?v=ckIIxKM14Ow)

- AlphaZero.jl - “Building on AlphaZero with Julia” by Jonathan Laurent: [Building on AlphaZero with Julia | Jonathan Laurent | JuliaCon2021 - YouTube](https://www.youtube.com/watch?v=nbLmR0aDumo)

- POMDPs.jl - A series of videos about “Decision Making Under Uncertainty using POMDPs.jl”: [Introduction to Decision Making Under Uncertainty using POMDPs.jl - YouTube](https://www.youtube.com/watch?v=yrC2O7T7aKU&list=PLP8iPy9hna6QPqk4N8eBk0oXzLrLbOtTA)

---

<div class="post-metadata">

**Author:** ![Lighfe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lighfe/32/38831_2.png) [@Lighfe](https://discourse.julialang.org/u/Lighfe)\
**Post date:** [August 16, 2022, 8:06pm UTC](https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836/5 "2022-08-16T20:06:30Z")

</div>

Thank you for the many resources. Would you recommend the books above “Reinforcement Learning - _An Introduction_”? Or rather as a go to for certain questions/problems that might occur?

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [August 16, 2022, 9:55pm UTC](https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836/6 "2022-08-16T21:55:01Z")

</div>

I guess that you are also addressing me in this question, however, I am very far from being an expert on those topics. I like all those books. I started with Sutton and Barto and I decided to send an email to @jonathan-laurent, however, it happened that he was busy. :- ) So I started digging deeper and I read this piece (not every line of course) by Kochenderfer, Wheeler and Wray and I decided to send an email to @findmyway. Slightly less, however, the same outcome, busy. :- ) So I came by this book by Bellemare, Dabney and Rowland. IMO, it outlines traditional and new reinforcement learning ideas really well, thus I think it might be a great choice as the first book. I’m also interested in quantum stuff a little bit and somehow I spotted some parallelism. This I think makes this book even more interesting, especially that the research on quantum reinforcement learning is so far rather limited. :- )

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [August 17, 2022, 1:30pm UTC](https://discourse.julialang.org/t/want-to-learn-reinforcement-learning-from-scratch-for-mini-project/85836/7 "2022-08-17T13:30:03Z")

</div>

I just recalled one more book that I liked (there are of course many others). Its not quite using Julia, however, the book is really written from a practical point of view. Hope you find the info that I tried to provide useful.

- “Deep Reinforcement Learning in Action”, Brandon Brown and Alexander Zai, Manning Publications Co. 2020 [Deep Reinforcement Learning in Action](https://www.manning.com/books/deep-reinforcement-learning-in-action).