Want to learn Reinforcement Learning from scratch for mini-project

[Lighfe]

Hello everyone,

I’m quite new to Julia but have some basic knowledge of other programming languages.

As a project for myself I want to train an RL algorithm for a boardgame. It’s a push-your-luck type of game, where you need to asses risk/rewards and therefore I think there are 100% dominant winning strategies which can be learned.

On the same time this seems like a nice opportunity to get more familiar with Julia.

I have theoretical background knowledge on machine learning but not RL specific. So I basically want to teach myself Reinforcement Learning from scratch as well as the Julia specifics.
Anybody has advice on how to approach this? I got quite some time on my hand during semester break. I just started reading “Reinforcement Learning: An Introduction” from Sutton & Barto, but I wonder how other people would approach this.

Thanks for any input/experiences

[digital_carver]

I don’t have experience implementing RL in Julia, but I’ll just share a few links I’ve come across:

• Getting Started With Reinforcement Learning in Julia episode on the Talk Julia podcast.
  The page also has a few useful Episode links at the bottom of the page, especially the An Introduction to ReinforcementLearning.jl: Design, Implementations and Thoughts blog post.

• Two videos on julia for talented amateurs on YouTube. This channel is usually light and easy to follow, so is a good way to take first steps into something. The videos are Markov Decision Process with POMDPs.jl and SARSA and Q-learning Algorithms with POMDPs.jl.

[albheim]

There is a repo implementing solutions for all problems in “Reinforcement Learning: An Introduction” using the library ReinforcementLearning.jl, so that could maybe be a good start.

There is also the library POMDPs.jl which also seem to cover many similar topics.

[j_u]

Hi Lighfe,

As for books there are:

• “Algorithms for Decision Making”; Mykel J. Kochenderfer, Tim A. Wheeler, Kyle H. Wray; The MIT Press 2022: https://mitpress.mit.edu/9780262047012/ AND https://algorithmsbook.com/

• “Distributional Reinforcement Learning”; Marc G. Bellemare and Will Dabney and Mark Rowland; The MIT Press 2023 (draft under submission): https://mitpress.mit.edu/9780262048019/distributional-reinforcement-learning/ AND https://www.distributional-rl.org/

There are interesting videos:

• ReinforcementLearning.jl - “A Practical Introduction to ReinforcementLearning.jl” by Jun Tian: ReinforcementLearning jl | Jun Tian | Julia User Group Munich - Share&Code - YouTube

• AlphaZero.jl - “Building on AlphaZero with Julia” by Jonathan Laurent: Building on AlphaZero with Julia | Jonathan Laurent | JuliaCon2021 - YouTube

• POMDPs.jl - A series of videos about “Decision Making Under Uncertainty using POMDPs.jl”: Introduction to Decision Making Under Uncertainty using POMDPs.jl - YouTube

[Lighfe]

Thank you for the many resources. Would you recommend the books above “Reinforcement Learning - An Introduction”? Or rather as a go to for certain questions/problems that might occur?

[j_u]

I guess that you are also addressing me in this question, however, I am very far from being an expert on those topics. I like all those books. I started with Sutton and Barto and I decided to send an email to @jonathan-laurent, however, it happened that he was busy. :- ) So I started digging deeper and I read this piece (not every line of course) by Kochenderfer, Wheeler and Wray and I decided to send an email to @findmyway. Slightly less, however, the same outcome, busy. :- ) So I came by this book by Bellemare, Dabney and Rowland. IMO, it outlines traditional and new reinforcement learning ideas really well, thus I think it might be a great choice as the first book. I’m also interested in quantum stuff a little bit and somehow I spotted some parallelism. This I think makes this book even more interesting, especially that the research on quantum reinforcement learning is so far rather limited. :- )

[j_u]

I just recalled one more book that I liked (there are of course many others). Its not quite using Julia, however, the book is really written from a practical point of view. Hope you find the info that I tried to provide useful.

• “Deep Reinforcement Learning in Action”, Brandon Brown and Alexander Zai, Manning Publications Co. 2020 Deep Reinforcement Learning in Action.