Reinforcement Learning

Reinforcement Learning (RL) is a paradigm where an agent learns to make decisions by interacting with an environment. The agent receives rewards or penalties and learns to maximize cumulative reward over time. RL powers game-playing AIs, robotics, and autonomous systems.

Exploration vs Exploitation

A key challenge in RL is balancing exploration (trying new actions to discover their effects) and exploitation (choosing known good actions). Epsilon-greedy is a common strategy where the agent explores with probability epsilon and exploits otherwise.

Q-Learning

Q-Learning is a model-free RL algorithm that learns the value of taking a given action in a given state. It uses a Q-table to store state-action values and updates them using the Bellman equation.

import numpy as np

# Simple Q-learning update
alpha = 0.1    # learning rate
gamma = 0.9    # discount factor

def q_learning_update(q_table, state, action, reward, next_state):
    best_next_action = np.argmax(q_table[next_state])
    td_target = reward + gamma * q_table[next_state][best_next_action]
    td_error = td_target - q_table[state][action]
    q_table[state][action] += alpha * td_error
    return q_table

SARSA

SARSA (State-Action-Reward-State-Action) is an on-policy RL algorithm. Unlike Q-Learning, it updates the Q-value based on the action actually taken, not the best possible action. This makes SARSA more conservative.

# SARSA update
def sarsa_update(q_table, state, action, reward, next_state, next_action):
    td_target = reward + gamma * q_table[next_state][next_action]
    td_error = td_target - q_table[state][action]
    q_table[state][action] += alpha * td_error
    return q_table

Actor-Critic Methods

Actor-Critic methods combine policy-based (actor) and value-based (critic) approaches. The actor decides which action to take, and the critic evaluates how good that action was. This approach often converges faster than pure policy or value methods.

Monte Carlo Methods

Monte Carlo methods learn from complete episodes. They wait until the end of an episode to update value estimates based on the total observed reward. They are unbiased but have high variance.

Temporal Difference Learning

Temporal Difference (TD) learning combines ideas from Monte Carlo and dynamic programming. It updates estimates based on other learned estimates (bootstrapping) without waiting for the episode to end. Q-Learning and SARSA are both TD methods.

Policy Gradients

Policy gradient methods directly optimize the policy by gradient ascent on expected reward. They work well for continuous action spaces and stochastic policies. REINFORCE is a classic policy gradient algorithm.

Environments (OpenAI Gym)

OpenAI Gym provides standardized environments for developing and comparing RL algorithms. Environments range from classic control (CartPole, MountainCar) to Atari games and robotics simulations.

import gymnasium as gym

# Create environment
env = gym.make('CartPole-v1')

# Run a random policy
observation, info = env.reset()
done = False
total_reward = 0

while not done:
    action = env.action_space.sample()  # random action
    observation, reward, terminated, truncated, info = env.step(action)
    total_reward += reward
    done = terminated or truncated

print(f"Total reward: {total_reward}")

Summary

Reinforcement learning enables agents to learn optimal behavior through trial and error. Key algorithms include Q-Learning, SARSA, Actor-Critic, and policy gradients. Practice with Gym environments to build your understanding.