Independent Learners (IQL, IDDPG)

The most direct way to apply reinforcement learning in a multi-agent setting is to simply let each agent learn independently, treating all other agents as part of the environment dynamics. This approach is often referred to as Independent Learning. It uses standard single-agent RL algorithms without modification, making it simple and easy to implement.

The Core Idea

Imagine you have $N$ agents in an environment. With independent learning, each agent $i \in \{1, ..., N\}$ maintains its own policy $\pi_i$ or value function (e.g., Q-function $Q_i$) and learns purely based on its own experiences. Agent $i$'s experience tuple typically consists of its local observation $s_i$, the action $a_i$ it took, the reward $r_i$ it received, and its next local observation $s'_i$.

During training, agent $i$ updates its policy or value function using a standard algorithm, completely ignoring the fact that the other agents $(j \neq i)$ are also learning and adapting their policies $\pi_j$.

Common Implementations: IQL and IDDPG

Two common instantiations of this idea are Independent Q-Learning (IQL) and Independent Deep Deterministic Policy Gradient (IDDPG).

Independent Q-Learning (IQL)

In IQL, each agent $i$ independently learns its own action-value function $Q_i(s_i, a_i)$ using the standard Q-learning update rule or its deep learning variant, DQN. If using tabular Q-learning, the update for agent $i$ would be:

$$Q_i(s_i, a_i) \leftarrow Q_i(s_i, a_i) + \alpha \left[ r_i + \gamma \max_{a'} Q_i(s'_i, a') - Q_i(s_i, a_i) \right]$$

Here, $s_i$ and $s'_i$ are agent $i$'s current and next states (or observations), $a_i$ is its action, $r_i$ is its reward, $\alpha$ is the learning rate, and $\gamma$ is the discount factor. If using neural networks (Independent DQN), each agent would have its own DQN network, trained using its own replay buffer filled with its $(s_i, a_i, r_i, s'_i)$ transitions. The loss function minimizes the difference between the target Q-value and the predicted Q-value, just as in single-agent DQN.

Independent Deep Deterministic Policy Gradient (IDDPG)

For environments with continuous action spaces, the DDPG algorithm can be applied independently by each agent. In IDDPG, each agent $i$ maintains its own actor network $\mu_i(s_i | \theta^{\mu_i})$ and critic network $Q_i(s_i, a_i | \theta^{Q_i})$. Each agent updates its networks based on its own experiences, using the standard DDPG updates. The actor aims to maximize the expected return predicted by its critic, while the critic learns to accurately estimate the Q-value for the actor's policy. Again, each agent treats the others as static parts of the environment during its update step.

Each agent learns independently, interacting with the environment and receiving its own observations and rewards. There is no direct sharing of learning updates or policy information between agents.

The Challenge: Non-Stationarity Strikes Back

The simplicity of independent learning comes at a significant cost. As mentioned in the chapter introduction, the primary challenge in MARL is non-stationarity. From the perspective of any single agent $i$, the environment appears non-stationary because the other agents $j \neq i$ are simultaneously updating their policies.

Consider agent $i$'s Q-learning update. The target value $r_i + \gamma \max_{a'} Q_i(s'_i, a')$ depends on the next state $s'_i$. However, $s'_i$ is determined not only by agent $i$'s action $a_i$ but also by the actions $a_j$ taken by all other agents $j$. Since the policies $\pi_j$ generating these actions $a_j$ are changing, the transition dynamics $P(s'_i | s_i, a_i)$ effectively change from agent $i$'s point of view.

This violates the fundamental Markov assumption that underlies Q-learning and many other RL algorithms. The assumption is that the environment's dynamics are stationary (fixed). When this assumption is broken:

1. Convergence Issues: Theoretical convergence guarantees for algorithms like Q-learning no longer hold. The learning process might oscillate, diverge, or converge to poor policies.
2. Unstable Targets: The target values used in updates (like the Bellman target in Q-learning or the critic target in DDPG) become unstable, making learning difficult. What seems like a good action now might become bad later simply because other agents changed their behavior.
3. Difficulty in Coordination: Since agents ignore each other's learning process, complex coordinated behavior is hard to achieve. Agents might learn conflicting strategies or fail to converge on mutually beneficial joint actions.

Advantages and Disadvantages Summarized

Advantages:

• Simplicity: Reuses existing single-agent RL algorithms directly. Easy to understand and implement.
• Lower Communication Needs (during execution): Once trained, agents typically only need their local observations to act.

Disadvantages:

• Non-Stationarity: The core problem, leading to unstable learning and potentially poor performance.
• Lack of Coordination: Agents don't explicitly model or account for others, hindering sophisticated cooperation or competition.
• Limited Applicability: Often struggles in tasks requiring tight coordination or where agents strongly influence each other's outcomes.

When to Use Independent Learners

Despite the significant drawback of non-stationarity, independent learning isn't entirely without merit. It can be effective in certain scenarios:

• Simple Environments: In tasks where agent interactions are minimal or the optimal policy doesn't require complex coordination.
• Baselines: It serves as a fundamental baseline to measure the performance improvement gained from more advanced MARL algorithms. If a sophisticated method can't outperform IQL or IDDPG, it might indicate issues with the method or that the problem doesn't require complex multi-agent reasoning.
• Resource Constraints: When implementation complexity or computational resources are limited.

However, for most complex multi-agent problems, the non-stationarity introduced by independent learning necessitates more specialized approaches. Techniques that explicitly address agent interactions, such as parameter sharing, centralized training with decentralized execution (CTDE), value decomposition methods, or multi-agent policy gradients, are often required to achieve stable and effective learning. These methods will be explored in the subsequent sections.