Reinforcement Learning Quick Reference

Quick reference guide for reinforcement learning algorithms, formulas, and code snippets.

Key Formulas

$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \ldots$$

$$Q(s,a) \leftarrow Q(s,a) + \alpha[r + \gamma \max_{a'} Q(s',a') - Q(s,a)]$$

$$\nabla_{\theta} J(\theta) = \mathbb{E}{\pi}[\nabla{\theta} \log \pi(a|s) \cdot Q^{\pi}(s,a)]$$

$$Q^{\pi}(s,a) = \sum_{s',r} P(s',r|s,a) [r + \gamma \sum_{a'} \pi(a'|s') Q^{\pi}(s',a')]$$

Algorithm	Type	Action Space	Off-Policy	Notes
Q-Learning	Value-based	Discrete	Yes	Simple, tabular
DQN	Value-based	Discrete	Yes	Deep, experience replay
Double DQN	Value-based	Discrete	Yes	Reduces overestimation
REINFORCE	Policy-based	Discrete/Continuous	No	Monte Carlo
A2C / typical Actor-Critic	Both	Discrete/Continuous	No	On-policy; learn from current policy rollouts
DDPG / SAC	Actor-Critic	Continuous	Yes	Off-policy actor-critic with replay
PPO	Policy-based	Discrete/Continuous	No	On-policy; clipped surrogate updates

Q[s, a] += alpha * (reward + gamma * np.max(Q[next_s]) - Q[s, a])

if random.random() < epsilon:
    action = random.choice(actions)
else:
    action = np.argmax(Q[state])

memory.append((state, action, reward, next_state, done))
batch = random.sample(memory, batch_size)

loss = -log_prob * advantage

Gymnasium: import gymnasium as gym; env = gym.make('CartPole-v1') then obs, info = env.reset() and obs, r, terminated, truncated, info = env.step(a)
Stable-Baselines3: from stable_baselines3 import DQN
Ray RLlib: from ray.rllib.algorithms import ppo