Reinforcement Learning

Deep Reinforcement Learning notes

In this section you can find our summaries from Sergey Levine (Google, UC Berkeley): UC Berkeley CS-285 Deep Reinforcement Learning course.

Basics

Introduction
  • Supervised vs Unsupervised vs Reinforcement
  • Ways to learn
  • How to build intelligent machines
  • State of the art
Reinforcement Learning Overview
  • Markov Decision Process
  • The goal of RL
  • RL algorithms
Imitation Learning
  • Behavioral Cloning
  • Why doesn't it work



Model-Free RL

Policy Gradients
  • Policy differentiation
  • The REINFORCE algorithm
  • Solving the causality issue
  • Baselines
  • Off-policy Policy Gradient
Actor-Critic (AC) Algorithms
  • Policy Gradients variance reduction
  • Policy Evaluation (Monte Carlo vs Bootstrapping)
  • Infinite horizon problems
  • Batch AC algorithm
  • Online AC algorithm
Value Function Methods
  • Policy Iteration
  • Value Iteration
  • Q iteration with Deep Learning
  • Q Learning
  • Exploration
Deep RL with Q-functions
  • Replay buffer and target network
  • DQN (Deep Q Networks)
  • Double Q-Learning
  • Multi-step returns
  • Continuous actions
  • DDPG (Deep Deterministic Policy Gradient)
Advanced Policy Gradients
  • Policy Gradient as Policy Iteration
  • The KL Divergence constraint
  • Dual Gradient Descent
  • Natural Gradients and Trust Region Policy Optimization
  • Proximal Policy Optimization



Model-Based RL

Model-based Planning
  • Deterministic vs Stochastic environments
  • Stochastic optimization methods
  • Monte Carlo Tree Search (MCTS)
  • Collocation trajectory optimization
  • Shooting trajectory optimization
Model-based Reinforcement Learning
  • Naive Model-Based RL
  • Uncertainty in model-based RL
  • Model-based RL with complex observations
Model-based Policy Learning
  • How to use env. models to learn policies
  • Local vs Global policies
  • Guided policy search
  • Policy Distillation
  • Divide & conquer RL



Advanced Topics

Inverse Reinforcement Learning
  • Underspecification problem
  • Feature Matching IRL
  • Maximum Entropy IRL
Transfer and Multi-task Learning
  • Forward Transfer
  • Multi-task Transfer
Meta-RL
  • Meta-Learning
  • Recurrent Meta-RL
  • Gradient-Based Meta-RL (MAML)
  • Meta-RL as a POMDP
  • Model-Based Meta-RL
Distributed RL
  • Original DQN
  • GORILA
  • A3C
  • IMPALA
  • Ape-X
  • R2D3
  • QT-Opt
  • Evolution Strategies
  • Population-based Training


Annex

This section contains both basic RL knowledge assumed to be known in the previous course and some demonstrations which we found interesting to add as an annex. In addition we added our own interpretations of some concepts hoping they can ease their understanding.

MDP Basics
Policy Expectations, Explained
Policy Gradients


Other great resources