Parameter Sharing Strategies

Treating each agent in a multi-agent system as a completely independent learner (like using Independent Q-Learning or IDDPG) is a straightforward approach. However, this method faces significant difficulties, particularly the non-stationarity problem. When many agents learn concurrently, the environment effectively changes from each agent's perspective, which destabilizes the learning process. Furthermore, training separate models for potentially numerous agents can be computationally expensive and data-inefficient, especially if the agents share similarities.

Parameter sharing offers a pragmatic strategy to mitigate some of these issues, particularly in scenarios involving homogeneous agents, meaning agents that have similar goals, observation spaces, and action spaces, or are interchangeable. The core idea is simple: instead of training entirely separate models (policy networks, value networks, or both) for each agent, we train a single model whose parameters are shared across multiple agents or even all agents.

Comparison between independent learners, each with its own set of model parameters ( $\theta_i$ ), and parameter sharing, where multiple agents utilize a single set of model parameters ( $\theta$ ).

How Parameter Sharing Works

In practice, during training, experiences (state transitions, actions, rewards) collected from multiple agents are used to update the single shared model. For instance, if using a shared policy network $\pi_{\theta}(a_i | o_i)$ for agent $i$ , the gradient update for the shared parameters $\theta$ would typically aggregate gradients calculated based on the experiences of all agents sharing those parameters.

\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} J_i(\theta)

Here, $J_i(\theta)$ is the objective function (e.g., policy gradient objective) computed using data from agent $i$ , and $N$ is the number of agents sharing the parameters $\theta$ . Each agent still receives its own observation $o_i$ and chooses its own action $a_i$ based on the shared policy conditioned on its specific observation. Similarly, if sharing a Q-network $Q_{\theta}(o_i, a_i)$ , updates would be based on transitions from all participating agents.

Advantages of Parameter Sharing

Improved Sample Efficiency: Since data from multiple agents contributes to training a single model, the model potentially learns faster and from less overall experience compared to training independent models. Each gradient step incorporates information from diverse interactions across different agents.
Reduced Computational Cost: Training and storing one model is generally less resource-intensive than managing numerous separate models, especially as the number of agents grows.
Potential for Better Generalization: The shared model might learn more generalizable features and behaviors applicable across different agents, especially if agents encounter slightly varied situations but operate under the same fundamental principles.
Scalability: It simplifies scaling to a large number of homogeneous agents without a linear increase in the number of models to train.

Limitations and Challenges

While attractive, parameter sharing isn't a universal solution. Its effectiveness depends on certain assumptions:

Agent Homogeneity: The most significant assumption is that agents are similar enough for a single policy or value function representation to be effective for all. If agents have drastically different capabilities, observation spaces, action spaces, or objectives, forcing them to share parameters can severely hinder performance. Imagine forcing a goalie and a striker in soccer to use the exact same policy network; it likely wouldn't work well.
Loss of Specialization: By definition, parameter sharing prevents agents from developing highly specialized individual behaviors that might be optimal in certain situations. The resulting policy is an average or compromise across the agents sharing the parameters.
Input Differentiation: Even with shared parameters, the network needs a way to differentiate between agents if their identities or specific contexts matter. Often, the agent's unique observation $o_i$ is sufficient. In some cases, a unique agent ID can be provided as an additional input to the shared network, allowing it to condition its output on the specific agent.

Variants and Extensions

Partial Parameter Sharing: Instead of sharing the entire network, specific layers (e.g., early layers for feature extraction) might be shared, while later layers remain agent-specific, offering a balance between efficiency and specialization.
Role-Based Parameter Sharing: In scenarios with distinct agent roles (e.g., defenders and attackers), agents within the same role might share parameters, while different roles have separate shared models.

Parameter sharing is often employed within the Centralized Training with Decentralized Execution (CTDE) framework, which we will discuss later. During centralized training, the shared parameters can be updated efficiently using aggregated data. During decentralized execution, each agent simply loads a copy of the trained shared model to make its decisions based on its local observations.

This technique represents a valuable tool in the MARL toolbox, particularly effective for cooperative tasks involving teams of similar agents, such as coordinating robot swarms or managing units in real-time strategy games. However, always examine the degree of agent homogeneity before applying it.

References

Multi-Agent Reinforcement Learning: A Review of Algorithms and Applications, Kaiqing Zhang, Zhuoran Yang, Tamer Başar, 2019 Foundations and Trends in Machine Learning, Vol. 12 (Now Publishers) DOI: 10.1561/2200000083 - This comprehensive review covers various MARL algorithms, including independent learners, CTDE, and discusses parameter sharing as a common technique, especially for homogeneous agents.
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments, Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch, 2017 Advances in Neural Information Processing Systems, Vol. 30 (NeurIPS) DOI: 10.5591/978-1-57783-000-8-124 - This influential paper introduces MADDPG, a CTDE framework, and highlights the non-stationarity problem of independent learning, providing context for why solutions like parameter sharing are needed. While not solely about parameter sharing, it's a prominent architecture where parameter sharing can be naturally applied for homogeneous agents within the CTDE paradigm.
Parameter Sharing with Role-Based Networks for Multi-Agent Reinforcement Learning, Arash Omidshafiei, Jason Ma, Michael A. Hutter, Shariq Iqbal, Jonathan P. How, 2019 Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33 (Association for the Advancement of Artificial Intelligence (AAAI)) DOI: 10.1609/aaai.v33i01.33014546 - This paper specifically explores variants of parameter sharing, such as role-based sharing and partial sharing, aligning directly with the 'Variants and Extensions' section.
Deep Reinforcement Learning Hands-On: Apply modern RL methods, with deep Q-networks, actor-critic models, and Beyond, Maxim Lapan, 2020 (Packt Publishing) - This practical guide offers implementations and discussions of various deep MARL techniques, including parameter sharing, making it valuable for readers looking for concrete examples and practical advice.