Limitations of Federated Averaging (FedAvg)

While Federated Averaging (FedAvg) provides a simple and intuitive baseline for federated learning, its effectiveness is often challenged in practical scenarios. Its limitations highlight the necessity of more sophisticated aggregation methods. FedAvg's core idea involves clients performing local stochastic gradient descent (SGD) updates and a central server averaging the resulting model parameters. This simplicity, however, hides several assumptions that frequently break down in practical deployments.

Let's examine the primary shortcomings:

Statistical Heterogeneity: The Non-IID Data Problem

Perhaps the most significant limitation of FedAvg stems from statistical heterogeneity, meaning the data distributions across clients are not independent and identically distributed (Non-IID). In many practical applications (like mobile keyboards predicting the next word for different users, or hospitals training models on their distinct patient populations), client data is inherently personalized and diverse.

Consider the global objective function that federated learning aims to minimize:

$$F(w) = \sum_{k=1}^N p_k F_k(w)$$

where $N$ is the number of clients, $p_k$ is the weight for client $k$ (often proportional to its dataset size), and $F_k(w)$ is the local objective function for client $k$ based on its local data $D_k$.

When data is Non-IID, the local optima $w_k^* = \arg\min_w F_k(w)$ can be significantly different for each client $k$, and also different from the global optimum $w^* = \arg\min_w F(w)$.

During local training in FedAvg, each client's model $w_k$ moves towards its local optimum $w_k^*$. If clients perform multiple local gradient descent steps (as is common in FedAvg to reduce communication), their models can diverge substantially from each other and from the current global model $w^t$. This phenomenon is often called client drift.

When the server averages these drifted local models ($w^{t+1} = \sum_{k \in S_t} p_k w_k^{t+1}$, where $S_t$ is the set of clients selected in round $t$), the resulting global model $w^{t+1}$ might not be a good compromise. It could perform poorly across all clients or even diverge. Simple averaging implicitly assumes that the average of local optima is close to the global optimum, which fails under significant Non-IID conditions.

Client models train towards local optima (colored circles) potentially far from the global optimum (star) due to Non-IID data. Averaging these drifted models may result in a suboptimal global model (orange circle).

This client drift due to Non-IID data leads to:

• Slower Convergence: The global model might oscillate or take many rounds to converge.
• Lower Final Accuracy: The converged global model might have significantly worse performance than a model trained on centralized data.
• Potential Divergence: In extreme cases, the training process might fail to converge altogether.

Systems Heterogeneity: Dealing with Diverse Clients

Federated networks often consist of clients with very different capabilities:

• Hardware: Varying CPU speeds, memory capacity, and potential accelerator availability (GPUs, TPUs).
• Network: Unstable or slow network connections (especially on mobile devices).
• Availability: Clients might join or leave the training process unpredictably.

Standard synchronous FedAvg, where the server waits for updates from a selected cohort of clients before performing aggregation, is particularly vulnerable to systems heterogeneity. The overall progress is dictated by the slowest client in each round (the "straggler" problem). This can severely slow down training, especially in large-scale deployments with diverse devices.

While asynchronous FL variants exist, simple averaging in these settings also faces challenges. Averaging models or updates computed based on different versions of the global model (staleness) without proper handling can degrade convergence and stability. Furthermore, if clients perform different amounts of local work (e.g., variable number of local epochs due to computational constraints) before sending updates, naive averaging might improperly weight their contributions.

Sensitivity to Malicious Updates (Byzantine Failures)

FedAvg treats all client updates equally during the averaging step. It assumes clients are honest and follow the protocol correctly. However, a malicious client (often called a Byzantine client) could intentionally send corrupted or poisoned updates designed to degrade the global model's performance or introduce backdoors.

Since FedAvg simply averages the parameters:

$$w^{t+1} = \frac{1}{|S_t|} \sum_{k \in S_t} w_k^{t+1} \quad \text{(unweighted version)}$$

a single client sending parameter values with extremely large magnitudes could completely dominate the average, effectively hijacking the global model. FedAvg lacks inherent robustness against such attacks. Even a small number of malicious participants can severely compromise the entire training process.

Communication Bottlenecks and Convergence Speed

While often cited as a benefit, communication efficiency in FedAvg still presents challenges. Transmitting full model updates (potentially millions of parameters for deep learning models) repeatedly can be costly, especially over constrained networks (like mobile connections). FedAvg aims to reduce the number of communication rounds compared to sending every gradient update, but the size of each communication payload remains large.

Furthermore, the convergence rate of FedAvg can be substantially slower than centralized training, particularly when dealing with high variance in client updates (due to sampling or Non-IID data) or when the number of local steps ($E$) is not carefully tuned. The averaging process itself is an approximation, and the variance introduced by client sampling and local updates can impede smooth convergence towards the global minimum.

These limitations highlight that while FedAvg laid the foundation, it often falls short in complex, federated settings. The subsequent sections in this chapter introduce advanced aggregation algorithms specifically designed to overcome these challenges, leading to more efficient and accurate federated learning systems. FedProx tackles client drift from Non-IID data, SCAFFOLD addresses variance and drift, FedNova handles systems heterogeneity, and Byzantine-robust methods provide resilience against malicious actors.