Building on our understanding of inference attacks, we now focus on techniques designed to uncover specific, often sensitive, characteristics of the data used to train a model. While membership inference asks if a particular record was in the training set, attribute inference attempts to determine the value of a specific attribute for a training record, even if that attribute wasn't directly used as input during inference.

Imagine a machine learning model trained to predict loan eligibility based on various applicant features. An attacker might possess partial information about an applicant (e.g., age, location, loan amount requested) and query the model. Based on the model's prediction (or its confidence score), the attacker might try to infer a sensitive attribute not initially known, such as the applicant's race or marital status, especially if these attributes were part of the original training data and correlated with the model's output.

The Goal of Attribute Inference

Formally, consider a data record $x$ from the training set $D_{train}$ . This record can often be split into public or known features $x_{pub}$ and one or more sensitive attributes $x_{sens}$ , so $x = (x_{pub}, x_{sens})$ . The attacker's objective is, given access to the model $f$ and the public features $x_{pub}$ of a target individual believed to be in the training set, to determine the value of $x_{sens}$ .

This attack poses a significant privacy risk because it allows adversaries to learn potentially private information about individuals whose data was used for training, leveraging the model as an information leakage channel.

Common Attribute Inference Strategies

Several approaches have been developed to perform attribute inference, often depending on the attacker's knowledge and the type of access they have to the target model.

1. Confidence-Based Attacks

One of the most intuitive approaches relies on the confidence scores produced by the model. The core idea is that a model might exhibit different confidence levels in its predictions depending on the value of the sensitive attribute, even if that attribute isn't directly fed into the model during prediction.

Suppose the attacker knows $x_{pub}$ for a target record $x$ . They want to determine if $x_{sens}$ equals value $v_1$ or value $v_2$ . The attacker can craft two hypothetical inputs: $x'_1 = (x_{pub}, x_{sens}=v_1)$ and $x'_2 = (x_{pub}, x_{sens}=v_2)$ . Although the attacker might not be able to directly query with $x_{sens}$ , they can observe the model's output $f(x)$ (where $x$ is the actual record with the unknown $x_{sens}$ ) or related outputs for similar known records.

The attacker hypothesizes that if the true sensitive attribute is $v_1$ , the model's output $f(x)$ might be "closer" (in some sense, perhaps prediction confidence) to the output $f(x'_1)$ than to $f(x'_2)$ . This often involves observing that models are sometimes more confident or make specific types of errors when presented with inputs characteristic of certain subgroups in the training data.

For example, an attacker might observe the confidence score of a facial recognition model identifying a person. If the model shows significantly higher confidence when the attacker assumes the person belongs to a certain demographic group (based on auxiliary information or statistical likelihood), they might infer that the person likely belongs to that group.

2. Likelihood Ratio Tests (LRT)

A more formal statistical approach involves using likelihood ratios. The attacker compares the likelihood of observing the model's output (e.g., prediction probabilities) given different possible values for the sensitive attribute.

Let $O$ be the observed output from the model $f$ when queried with information related to the target record (often using $x_{pub}$ ). The attacker wants to decide between two hypotheses: $H_1: x_{sens} = v_1$ and $H_2: x_{sens} = v_2$ . The likelihood ratio is:

LR = \frac{P(O | H_1)}{P(O | H_2)}

If $LR > 1$ , the observed output is more likely under the assumption that $x_{sens} = v_1$ . If $LR < 1$ , it's more likely that $x_{sens} = v_2$ . Calculating these conditional probabilities $P(O | H_i)$ usually requires the attacker to have some background knowledge or to build a model of how $f$ 's outputs correlate with $x_{sens}$ .

3. Using an Attack Model

Similar to shadow training in membership inference, an attacker can train a dedicated attack model specifically for attribute inference.

The process typically involves these steps:

Obtain Data: The attacker needs data points $(x'_{pub}, f(x'), x'_{sens})$ where $x'_{pub}$ are public features, $f(x')$ is the target model's output (e.g., prediction vector) for a record $x'$ , and $x'_{sens}$ is the known sensitive attribute for that record. This data might come from public datasets, auxiliary sources, or by synthesizing queries if the attacker has partial knowledge of the training distribution.
Train Attack Model: Train a classifier, the attack model $g_{attack}$ , that takes the public features $x'_{pub}$ and the target model's output $f(x')$ as input, and predicts the sensitive attribute $x'_{sens}$ . $g_{attack}(x'_{pub}, f(x')) \approx x'_{sens}$
Infer Attribute: For a target record $x$ with known $x_{pub}$ and observed target model output $f(x)$ , the attacker uses their trained model to predict the sensitive attribute: $\hat{x}_{sens} = g_{attack}(x_{pub}, f(x))$ .

The success of this method depends heavily on the quality and relevance of the data used to train the attack model and the correlation between the target model's outputs and the sensitive attribute.

An attacker trains an attack model using known public features and corresponding outputs from the target model to predict unknown sensitive attributes for new target records.

Factors Influencing Attribute Inference Success

The effectiveness of attribute inference attacks depends on several factors:

Correlation: The stronger the correlation between the sensitive attribute $x_{sens}$ and the features used by the model or the model's output $f(x)$ , the easier the inference. If an attribute has minimal influence on the model's predictions, inferring it is much harder.
Model Overfitting: Models that overfit to the training data may inadvertently memorize specific correlations tied to sensitive attributes, making them more vulnerable.
Attacker's Knowledge: The amount and quality of auxiliary information available to the attacker significantly impact their ability to build effective attack models or calculate likelihood ratios.
Output Granularity: Access to detailed outputs like confidence scores or probability vectors generally provides more information for inference than just final class labels.

Attribute Inference Example: Inferring Location

Consider a model trained on user posts to classify sentiment (positive/negative). The training data includes the post text (used as input) and metadata like user location (potentially sensitive). An attacker might have access to a user's posts ( $x_{pub}$ ) and the model's sentiment predictions $f(x)$ . If users from certain locations tend to use specific phrasing or discuss local topics that influence sentiment prediction, the attacker could potentially train an attack model $g_{attack}(\text{posts}, f(\text{posts}))$ to infer the user's location ( $x_{sens}$ ).

Example distribution showing how model confidence might differ based on an underlying sensitive attribute. High confidence predictions are more frequent when the attribute is B, potentially allowing an attacker to infer the attribute by observing confidence scores.

Relationship to Privacy and Defenses

Attribute inference is fundamentally a privacy attack. It highlights that even if sensitive data isn't directly requested or output by a model, the model's behavior can still leak this information. Techniques like differential privacy aim to provide formal guarantees against such leakage by ensuring that the model's output doesn't change significantly whether or not any single individual (and their attributes) is included in the training set. Regularization methods that reduce overfitting can also indirectly help mitigate attribute inference by preventing the model from memorizing spurious correlations tied to sensitive attributes. We will discuss defenses against inference attacks, including differential privacy, more thoroughly in Chapter 5.

Understanding attribute inference is essential for assessing the real-world privacy implications of deploying machine learning models, particularly those trained on sensitive personal data. It compels us to consider not just model accuracy but also the potential for information leakage through model interactions.