Deep & Quantum ML

Deep learning extends traditional neural networks with many layers, enabling models to learn hierarchical representations from raw data. When combined with reinforcement learning or quantum computing concepts, it opens frontiers in AI research and applications.

Deep Q-Networks (DQN)

DQN combines Q-learning with deep neural networks to handle high-dimensional state spaces like images. The neural network approximates the Q-value function, mapping states to action values. Key innovations include experience replay (storing past experiences for training) and target networks for stable learning.

import numpy as np

class DQNAgent:
    def __init__(self, state_size, action_size):
        self.state_size = state_size
        self.action_size = action_size
        self.memory = []
        self.gamma = 0.95
        self.epsilon = 1.0
        self.epsilon_min = 0.01
        self.epsilon_decay = 0.995

    def remember(self, state, action, reward, next_state, done):
        self.memory.append((state, action, reward, next_state, done))

    def act(self, state):
        if np.random.rand() <= self.epsilon:
            return np.random.choice(self.action_size)
        # In practice, use model.predict(state) here
        return np.argmax(state)

    def replay(self, batch_size):
        # Sample from memory and train the model
        if len(self.memory) < batch_size:
            return
        # Training logic with target network would go here
        if self.epsilon > self.epsilon_min:
            self.epsilon *= self.epsilon_decay

Policy Gradient Methods

Unlike value-based methods (DQN), policy gradient methods directly optimize the policy — the mapping from states to actions. They work well in continuous action spaces and learn stochastic policies, which are often better for exploration.

import numpy as np

def policy_gradient_update(states, actions, rewards, policy_probs, lr=0.01):
    # Simplified policy gradient update
    discounted_rewards = []
    cumulative = 0
    for r in reversed(rewards):
        cumulative = r + 0.99 * cumulative
        discounted_rewards.insert(0, cumulative)

    discounted_rewards = np.array(discounted_rewards)
    discounted_rewards = (discounted_rewards - np.mean(discounted_rewards)) / \
                         (np.std(discounted_rewards) + 1e-8)

    # Gradient: log_prob * advantage
    for state, action, reward, prob in zip(states, actions, discounted_rewards, policy_probs):
        # Simplified: theta += lr * reward * grad(log(pi(a|s)))
        pass

Trust Region Policy Optimization (TRPO)

TRPO improves the stability of policy gradient methods by constraining the update step size. It ensures the new policy does not deviate too far from the old policy by using a KL-divergence constraint, preventing catastrophic performance collapses during training.

Proximal Policy Optimization (PPO)

PPO simplifies TRPO by using a clipped surrogate objective instead of a KL constraint. It limits the policy update ratio to a small range (e.g., [0.8, 1.2]), making it easier to implement while maintaining training stability. PPO has become the default algorithm in many RL applications.

import numpy as np

def ppo_clip_loss(old_log_prob, new_log_prob, advantage, epsilon=0.2):
    ratio = np.exp(new_log_prob - old_log_prob)
    clipped_ratio = np.clip(ratio, 1 - epsilon, 1 + epsilon)
    return -np.minimum(ratio * advantage, clipped_ratio * advantage)

# Example advantage computation
advantages = np.random.randn(64)
old_probs = np.random.rand(64)
new_probs = np.random.rand(64)
loss = ppo_clip_loss(old_probs, new_probs, advantages)
print(f"PPO clip loss: {np.mean(loss):.4f}")

Neural Network Architectures for RL

Deep RL relies on various neural network architectures: Convolutional Neural Networks (CNNs) for visual inputs, Recurrent Neural Networks (RNNs) with LSTM or GRU units for partial observability and temporal dependencies, and Transformers for long-horizon memory and attention-based decision making.

Quantum ML Concepts Overview

Quantum Machine Learning explores how quantum computing can accelerate or improve ML tasks. Key ideas include using quantum circuits as neural networks, quantum kernels for SVM, and variational quantum algorithms that combine classical optimization with quantum computation. While still experimental, QML promises exponential speedups for specific problems.