Efficient Representations are Controllable Representations

What is the most brute-force way to install interpretable, controllable features into a model’s activations? Controlling how LLMs internally represent concepts typically requires sophisticated methods to first identify, then intervene on the model’s existing feature geometry. We bypass all of this. We finetune an LLM with a simple auxiliary loss, training 16 of its 3072 residual stream dimensions to be inert interpretability flags that simply indicate what concepts are required for generation. The model reorganizes around them anyway, learning to rely on these flags during actual generation tasks. As a result, these inert flags become genuine internal features: interpretable control switches that allow us to steer generation at inference time. Why does this work? When a feature is reliably supplied at a fixed location, gradient descent gradually eliminates redundant encodings elsewhere, and the model erodes its own alternative representations. A model’s efficiency pressure is a lever - exploitable to induce interpretable, controllable representations.

💡 Research Summary

The paper tackles the longstanding problem of making large language models (LLMs) both interpretable and controllable. Traditional mechanistic interpretability follows a two‑step pipeline: first, discover latent feature directions using probes, sparse autoencoders, or representation‑engineering techniques; second, intervene on those directions to steer model behavior. This approach is hampered by superposition, polysemanticity, and the fact that interventions are limited to the geometry the model has already learned.

Instead of decoding the existing geometry, the authors propose a brute‑force method that installs a small, fixed set of interpretable “flags” directly into the residual stream of a pretrained LLM. Using Phi‑3‑Mini (3.8 B parameters, hidden dimension 3072), they reserve 16 dimensions (out of 3072) across all layers—four dimensions per concept for five concepts (dogs, cats, animals, food, programming). These dimensions receive no special routing, attention heads, or architectural changes; they are simply designated as binary classifiers that should be high when the corresponding concept is active and near‑zero otherwise.

Training proceeds in two stages. In stage 1, the correct binary labels are injected directly into the flagged dimensions at every layer, giving the model perfect feature information for free. This forces the downstream computation to learn to use the flags without having to generate them. In stage 2, the injections are removed and a position loss (mean‑squared error between the model’s actual values in the flagged dimensions and the target binary values) is added to the standard cross‑entropy language‑modeling loss. The weight λₜ of the position loss is gradually increased, shifting pressure from “learn to read the flags” to “learn to produce them”. The entire model (all 3.8 B parameters) is fine‑tuned on ~100 M synthetic tokens generated from auxiliary LLMs, each text being randomly assigned a subset of the five concepts. The labels indicate whether the generation requires knowledge of a concept, not merely whether the word appears.

Results show three key phenomena. First, after training the flagged dimensions reliably activate for the appropriate concepts, especially from layer 5 onward, confirming that the model has learned to write interpretable signals into the reserved slots. Second, at inference time the authors overwrite the flagged dimensions with arbitrary binary patterns. This deterministic manipulation dramatically changes the model’s output: forcing “dogs + animals” yields a dog story, forcing “animals + dogs off” yields an animal story without dogs, and even forcing “programming” (which is not mentioned in the prompt) causes the model to generate a coding narrative. The model therefore not only writes the flags but also reads them downstream, using them as authoritative cues for subsequent computation. Third, to test whether the model has truly consolidated the information into the flags (and discarded redundant copies), linear probes are trained to predict concept presence. Probes on the original model use all 3072 dimensions; probes on the fine‑tuned model are restricted to the 3056 non‑flagged dimensions. Accuracy drops by 4–18 percentage points for each flagged concept, while an unflagged control (finance) shows only a 1.6 point drop. This indicates that the model has migrated most of the concept information into the flagged dimensions, eroding its previous distributed encodings.

The authors interpret these findings through the lens of “efficiency pressure”. In a finite‑capacity residual stream, maintaining duplicate representations is wasteful. Providing a reliable, fixed‑location signal (the flags) creates a cheap source of information; the language‑modeling objective then incentivizes the model to reclaim the freed dimensions for other work, leading to consolidation around the flags. The paper also quantifies the cost of commandeering more dimensions: perplexity on WikiText‑2 rises modestly from 11.04 (baseline) to 11.16 with 16 flagged dimensions, but increases to 12.04 with 64 dimensions and 12.22 with 128 dimensions, illustrating a trade‑off between controllability and raw predictive performance.

Related work is discussed in two categories. Post‑hoc interpretability methods (probes, SAEs, activation addition) are powerful but constrained by the existing geometry; intrinsic interpretability approaches (concept bottleneck models, self‑explaining networks) impose architectural bottlenecks that limit capacity. The present method imposes no bottleneck—3056 dimensions remain free—yet the model voluntarily adopts the flags because of efficiency considerations.

In the discussion, the authors argue that the principle uncovered is general: any training signal that reliably supplies a feature at a fixed location should cause a model to consolidate around it, regardless of implementation details. Consequently, the traditional framing of controllability (“decode then intervene”) may be inverted to “provide reliable signals and let the optimizer do the rest”. The paper demonstrates that the optimizer is a collaborator: by setting up the right conditions (fixed‑location, low‑cost signals), models will naturally turn crude interventions into genuine, writable internal structure. This insight opens a new avenue for building controllable LLMs without complex architectural modifications or exhaustive feature‑discovery pipelines.