Value Decomposition Methods (VDN, QMIX)

Maximizing a shared team reward is a common goal in cooperative multi-agent settings. The Centralized Training with Decentralized Execution (CTDE) approach offers a practical method: agents learn together using potentially global information but must act independently at execution time based only on their local observations. A primary challenge within CTDE is coordinating actions effectively. When a centralized joint action-value function, $Q_{tot}(\mathbf{s}, \mathbf{a})$, is learned (where $\mathbf{s}$ is the global state or some representation available during training, and $\mathbf{a}$ is the joint action $(a_1, ..., a_N)$), a primary question is how to extract decentralized policies $\pi_i(a_i | o_i)$ that agents can use based on their local observation $o_i$.

Performing a global argmax over $\mathbf{a}$ for $Q_{tot}$ is often computationally intractable, especially as the number of agents and actions grows. Value Decomposition Methods address this by learning individual agent value functions $Q_i(o_i, a_i)$ and then combining them in a structured way to approximate $Q_{tot}$, ensuring that maximizing $Q_{tot}$ can be achieved by each agent maximizing its own $Q_i$.

Value Decomposition Network (VDN)

The simplest approach to value decomposition is the Value Decomposition Network (VDN). VDN assumes that the joint action-value function can be additively decomposed into individual agent value functions:

$$Q_{tot}(\mathbf{s}, \mathbf{a}) = \sum_{i=1}^{N} Q_i(o_i, a_i)$$

Here, $N$ is the number of agents. Each $Q_i$ is typically represented by a neural network that takes agent $i$'s local observation $o_i$ and action $a_i$ as input (often, the network outputs Q-values for all possible actions $a_i$ given $o_i$). The global $Q_{tot}$ is simply the sum of these individual Q-values.

How it enables CTDE:

1. Centralized Training: During training, we can compute $Q_{tot}$ using the sum of individual $Q_i$ values. The loss (e.g., TD error from Q-learning) is calculated based on $Q_{tot}$ and the global reward, and gradients are backpropagated through the sum operation to update each individual $Q_i$ network. This often requires access to the global state $\mathbf{s}$ or at least the joint action $\mathbf{a}$ and global reward $r$.
2. Decentralized Execution: At execution time, each agent $i$ only needs its local observation $o_i$. It computes its $Q_i(o_i, \cdot)$ for all its possible actions and chooses the action greedily: $$a_i^* = \arg\max_{a_i} Q_i(o_i, a_i)$$ Because the global $Q_{tot}$ is just a sum, maximizing each $Q_i$ individually guarantees that the joint action $\mathbf{a}^* = (a_1^*, ..., a_N^*)$ maximizes $Q_{tot}$.

VDN architecture. Each agent has an independent Q-network. During training, their outputs are summed to form Q_tot for loss calculation. During execution, each agent acts greedily based on its own Q_i.

Limitations: The main limitation of VDN is its restrictive assumption of additivity. It can only represent joint action-value functions where the contribution of each agent is independent of the others' actions. This prevents it from modeling more complex coordination scenarios where the value of one agent's action heavily depends on what other agents are doing.

QMIX: Monotonic Value Function Factorization

QMIX (Q-Mixing) addresses the representational limitations of VDN while maintaining the convenient decentralized execution property. Instead of a simple summation, QMIX uses a mixing network to combine the individual $Q_i$ values into $Q_{tot}$.

$$Q_{tot}(\mathbf{s}, \mathbf{a}) = f(\{Q_i(o_i, a_i)\}_{i=1}^N, \mathbf{s})$$

The function $f$, represented by the mixing network, is designed to satisfy an important monotonicity constraint:

$$\frac{\partial Q_{tot}}{\partial Q_i} \ge 0 \quad \forall i$$

This constraint ensures that if an agent increases its individual $Q_i$ value, the global $Q_{tot}$ value will either increase or stay the same, but never decrease.

Why Monotonicity Matters: The monotonicity constraint is sufficient to guarantee that a global argmax over $Q_{tot}$ yields the same result as performing individual argmax operations on each $Q_i$:

$$\arg\max_{\mathbf{a}} Q_{tot}(\mathbf{s}, \mathbf{a}) = (\arg\max_{a_1} Q_1(o_1, a_1), ..., \arg\max_{a_N} Q_N(o_N, a_N))$$

This means QMIX retains the ease of decentralized execution found in VDN, where each agent simply picks the action maximizing its own learned $Q_i$.

Mixing Network Architecture: The mixing network enforces the monotonicity constraint by having non-negative weights. It typically takes the outputs of the individual $Q_i$ networks as input. Crucially, the weights and biases of the mixing network itself are generated by separate hypernetworks that receive the global state $\mathbf{s}$ as input. This allows the way the individual $Q_i$ values are combined to depend on the overall context provided by the state, making QMIX much more expressive than VDN. The non-negativity of the weights is usually enforced by using an absolute activation function or ReLU on the output of the weight-generating hypernetworks.

QMIX architecture. Individual agent Q-networks ($Q_1$ to $Q_N$) operate decentrally. Their outputs feed into a central mixing network. The mixing network's parameters (weights and biases) are generated by hypernetworks conditioned on the global state $s$, ensuring monotonicity ($\partial Q_{tot}/\partial Q_i \ge 0$) and allowing complex, state-dependent combinations while preserving tractable decentralized execution.

Advantages over VDN: QMIX can represent a much richer class of cooperative MARL problems than VDN because the mixing network can learn complex non-linear combinations of the individual agent values, conditioned on the global state. The only structural constraint is monotonicity, which is significantly less restrictive than pure additivity.

Summary: VDN and QMIX are prominent value-based methods within the CTDE framework for cooperative MARL. They learn individual agent Q-functions $Q_i$ that allow for decentralized execution.

• VDN uses a simple sum, making it easy to implement but limiting its representational power.
• QMIX employs a state-dependent mixing network with a monotonicity constraint, offering greater expressiveness while retaining the critical property that maximizing individual $Q_i$ functions leads to maximization of the joint $Q_{tot}$.

These methods provide effective ways to coordinate teams of agents by factorizing the team's value function, forming a foundation for many advanced techniques in cooperative multi-agent reinforcement learning.