Threat Models in Federated Learning

Understanding the potential vulnerabilities of a federated learning system is essential before designing defenses. A threat model defines the capabilities and goals of potential adversaries, helping to analyze risks and evaluate countermeasures. The distributed nature of federated learning introduces unique attack surfaces compared to traditional centralized machine learning.

We typically categorize threats based on the adversary's position within the system and their objectives. Adversaries might be malicious clients participating in the training, the central server coordinating the process, or even external eavesdroppers intercepting communication. Their goals can range from degrading the global model's performance to extracting sensitive information about participants' private data.

Adversary Locations and Capabilities

Let's examine the primary locations from which an adversary might operate:

Malicious Clients (Insider Threats)

Clients participating in the FL process are inherently trusted to follow the protocol. However, some clients might be malicious or compromised. These "insider" adversaries control their local data and computation. Their capabilities typically include:

• Data Manipulation: They can alter their local dataset before training begins.
• Computation Manipulation: They can deviate from the prescribed local training algorithm (e.g., modify gradients, change local objective function).
• Update Manipulation: They can craft malicious model updates or gradients to send back to the server.

A common assumption is that the adversary controls a fraction of the total clients, often denoted by $f$. The adversary might coordinate these malicious clients (collusion) or act independently.

Malicious Server/Aggregator

The central server, while not having direct access to raw client data, orchestrates the FL process. A compromised or malicious server possesses significant power:

• Access to Aggregated Updates: The server observes all updates (gradients or model weights) sent by clients, although privacy techniques might encrypt or obscure them.
• Control over Aggregation: The server implements the aggregation logic and could potentially manipulate it.
• Model Distribution: The server disseminates the updated global model to clients.
• Client Selection: The server often selects which clients participate in each round.

A common model here is the "honest-but-curious" server. This server follows the FL protocol correctly but attempts to infer information from the legitimate updates it receives. A fully malicious server might actively tamper with the process.

External Eavesdropper

An attacker outside the core FL system (clients and server) might attempt to intercept communication between clients and the server.

• Observation of Updates: If communication channels are insecure, an eavesdropper could observe the model updates being transmitted. The sensitivity of this depends on whether the updates themselves are protected (e.g., via encryption or secure aggregation).

Modern FL systems often assume secure communication channels (e.g., TLS/SSL), making passive eavesdropping on raw updates less of a primary concern compared to threats from malicious participants or the server itself. However, metadata leakage (e.g., timing, frequency of updates) might still be possible.

Adversaries can exist as malicious clients, a compromised server, or external eavesdroppers, each targeting different parts of the FL process.

Adversary Goals and Attack Types

Based on their capabilities and position, adversaries pursue different goals:

Poisoning Attacks (Integrity and Availability)

Primarily orchestrated by malicious clients, poisoning attacks aim to corrupt the training process or the final global model.

• Data Poisoning: Malicious clients manipulate their local training data to skew the local model updates they compute. For example, they might mislabel data points or inject samples designed to cause misclassification for specific inputs later. This influences the global model indirectly after aggregation.
• Model Poisoning: Malicious clients directly manipulate the model updates they send to the server, without necessarily modifying their underlying data. This provides more direct control over the attack.
  • Untargeted Attacks: Aim to degrade the overall performance of the global model on its primary task (Availability attack). The goal is simply to make the model less accurate or useless.
  • Targeted Attacks (Backdoor Attacks): A more sophisticated attack where the goal is to cause the global model to misbehave only on specific, adversary-chosen inputs (e.g., classify images with a specific watermark as a certain category), while maintaining good performance on general data. This compromises the model's integrity.

Detecting and mitigating poisoning attacks is challenging, especially in heterogeneous (Non-IID) settings where deviating updates might resemble legitimate updates from clients with unusual data distributions. Aggregation rules, discussed in Chapter 2, are a primary defense mechanism.

Inference Attacks (Privacy Violation)

These attacks aim to extract sensitive information about clients' private data, often perpetrated by a curious server or potentially other clients or eavesdroppers if updates are not properly secured. Since raw data ideally never leaves the client device, attackers try to infer information from the shared model updates (gradients or weights).

• Membership Inference: Attempting to determine whether a specific data record was part of a particular client's training dataset. Knowing that someone's record was used in training (e.g., for a medical model) can itself be sensitive.
• Property Inference: Trying to deduce aggregate properties of a client's dataset that were not explicitly intended to be shared (e.g., the proportion of a certain demographic group in the training data, even if individual records aren't identified).
• Data Reconstruction/Gradient Inversion: The most potent privacy attack, aiming to reconstruct approximations of the actual training data samples from the shared gradients or model updates. While exact reconstruction is difficult, even partial reconstruction can leak significant information, especially for high-dimensional data like images or text. Techniques often involve optimizing input data to match the observed gradients, sometimes leveraging knowledge of the model architecture or properties of the data distribution.

These attacks highlight that simply not sharing raw data is insufficient for privacy. The model updates themselves carry information that can be exploited. Privacy-enhancing techniques like Differential Privacy (DP) and Secure Multi-Party Computation (SMC), covered in detail in Chapter 3, are designed to formally limit such information leakage.

Understanding these threat models is fundamental. When designing or analyzing advanced FL techniques (like new aggregation rules, privacy mechanisms, or communication strategies), we must constantly evaluate their resilience against these potential attacks. The assumptions made about the adversary (e.g., their computational power, knowledge of the system, fraction of controlled clients) significantly influence the effectiveness and applicability of different defense mechanisms.