While feature attribution methods help us understand which parts of the input influenced a specific output, concept probing and representation analysis provide a different lens into the model's internal workings. These techniques aim to determine if and where abstract concepts, particularly those relevant to safety like toxicity, bias, or honesty, are encoded within the model's intermediate representations (activations). Understanding this can be significant for diagnosing unwanted behaviors and verifying alignment mechanisms.

Instead of focusing solely on the input-output relationship for a single instance, probing investigates whether consistent patterns emerge in the LLM's activation space across many inputs related to a specific concept.

The Probing Paradigm

The core idea is relatively straightforward: train a simple auxiliary model, known as a "probe," to predict the presence or properties of a concept based only on the internal activations of the LLM.

Choose a Concept: Define the concept you want to investigate. This could be a binary property (e.g., "is toxic" vs. "is not toxic"), a categorical one (e.g., "topic is politics/sports/science"), or even a continuous value (e.g., "sentiment score"). Safety-related concepts often include toxicity, bias dimensions (gender, race), honesty, or adherence to specific safety instructions.
Select Target Representations: Decide which internal activations of the LLM will serve as input to the probe. This typically involves choosing specific layers (e.g., intermediate layers of the transformer blocks, the final layer before the output logits) or specific components (e.g., outputs of attention heads or MLP sublayers).
Gather Data: Collect a dataset where each instance includes:
- An input text fed to the LLM.
- The corresponding internal activations from the chosen layer(s) of the LLM.
- A label indicating the presence, absence, or value of the concept for that input. For example, inputs could be sentences labeled as 'toxic' or 'non-toxic'.
Train the Probe: Train a probe model (often a simple one, like a linear classifier or a small multi-layer perceptron, MLP) to predict the concept label using the LLM's activations as input features.
Evaluate the Probe: Assess the probe's accuracy. If the probe performs significantly better than chance, it suggests that information about the concept is indeed encoded, or at least linearly decodable, within the chosen LLM representations.

A simplified view of the concept probing setup. Activations from a specific layer of a frozen LLM are used to train a separate probe model to predict a predefined concept label.

Types of Probes and Interpretation

The choice of probe model carries implications:

Linear Probes: A logistic regression or linear Support Vector Machine (SVM) is often used. High accuracy with a linear probe suggests the concept is linearly represented in the activation space. This is a strong indicator that the model explicitly encodes the concept in a straightforward geometric way. The weights of the linear probe can sometimes be interpreted as a "concept vector" within the activation space.
Non-Linear Probes: Small MLPs can capture more complex, non-linear relationships between activations and the concept. While potentially more accurate, a highly accurate non-linear probe makes interpretation harder. It might indicate the concept information is present but entangled in a complex way, or the probe itself might be learning the classification task using the rich activation features without necessarily implying a simple encoding within the LLM itself. There's a trade-off between probe accuracy and the clarity of the claim about the LLM's internal representation.

Analyzing Representations Beyond Classification

Probing isn't limited to classification. Representation analysis techniques delve deeper:

Directional Probing: Instead of just classifying presence/absence, probes can be trained to predict continuous values (e.g., toxicity score). The gradient of the probe's prediction with respect to the activations, or the weights of a linear probe, can suggest directions in the activation space that correspond to increasing or decreasing intensity of the concept.
Comparative Analysis: Probes can be trained for related concepts (e.g., different types of bias, or toxicity vs. formality) on the same representations. Comparing the probe weights or performance can reveal whether the LLM distinguishes between these concepts internally or conflates them.
Layer-wise Evolution: By training probes on activations from different layers, you can trace how the representation of a concept evolves as information flows through the LLM. Safety-relevant concepts might only become clearly distinguishable in later layers.

Applications in Safety

Concept probing is particularly valuable for safety:

Identifying Bias Encoding: Training probes to detect gender, race, or other demographic attributes based on activations when processing neutral prompts can reveal unwanted stereotypical associations encoded within the model. For instance, does the activation space differ systematically when processing sentences about "doctors" versus "nurses" even if the context is identical?
Detecting Latent Harm: Can we train a probe to predict if the model is about to generate harmful content based on its internal state, even before the tokens are emitted? This could potentially inform faster, internal safety mechanisms.
Verifying Safety Training: If a model has undergone safety fine-tuning (like RLHF or Constitutional AI), probes can be used to check if concepts related to the safety instructions (e.g., "refusal," "harmlessness") are now more clearly represented or separable in the activation space compared to the base model.
Understanding Failure Modes: When a model fails a safety benchmark or generates an undesirable output despite alignment, probing the activations during that generation can help diagnose why. Was the representation of the harmful concept too similar to a harmless one? Did the safety mechanism's representation fail to activate strongly?

Challenges and Important Considerations

While powerful, probing requires careful interpretation:

Correlation is Not Causation: A successful probe demonstrates that concept information is decodable from the activations. It doesn't necessarily mean that these specific activations cause the model's behavior related to the concept or that the model "uses" this information in the way the probe does.
Probe Faithfulness: The probe should ideally leverage the same information the LLM uses. A complex probe might learn the task using subtle correlations unrelated to the LLM's primary mechanism for representing the concept. Simpler probes are generally preferred for interpretability.
Concept Definition and Data Quality: The results are highly dependent on how well the concept is defined and how accurately the dataset is labeled. Probing for vague concepts like "fairness" is significantly harder than probing for concrete categories like "mentions a specific location."
Computational Cost: Extracting activations and training probes, especially across many layers and concepts for large models, requires substantial computational resources.

Concept probing and representation analysis offer a valuable window into the internal state of LLMs, complementing other interpretability methods. By examining how safety-relevant concepts are encoded (or not encoded) within the model's hidden states, we gain deeper insights crucial for building more reliable and verifiable AI systems. These techniques move beyond surface-level behavior analysis, allowing for a more direct assessment of whether a model has truly internalized the principles underlying its safety training.