Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs

Developing human understandable interpretation of large language models (LLMs) becomes increasingly critical for their deployment in essential domains. Mechanistic interpretability seeks to mitigate the issues through extracts human-interpretable process and concepts from LLMs’ activations. Sparse autoencoders (SAEs) have emerged as a popular approach for extracting interpretable and monosemantic concepts by decomposing the LLM internal representations into a dictionary. Despite their empirical progress, SAEs suffer from a fundamental theoretical ambiguity: the well-defined correspondence between LLM representations and human-interpretable concepts remains unclear. This lack of theoretical grounding gives rise to several methodological challenges, including difficulties in principled method design and evaluation criteria. In this work, we show that, under mild assumptions, LLM representations can be approximated as a {linear mixture} of the log-posteriors over concepts given the input context, through the lens of a latent variable model where concepts are treated as latent variables. This motivates a principled framework for concept extraction, namely Concept Component Analysis (ConCA), which aims to recover the log-posterior of each concept from LLM representations through a {unsupervised} linear unmixing process. We explore a specific variant, termed sparse ConCA, which leverages a sparsity prior to address the inherent ill-posedness of the unmixing problem. We implement 12 sparse ConCA variants and demonstrate their ability to extract meaningful concepts across multiple LLMs, offering theory-backed advantages over SAEs.

💡 Research Summary

The paper tackles the long‑standing problem of linking the hidden states of large language models (LLMs) to human‑understandable concepts. It begins by positing a discrete latent‑variable generative model in which observed text (the context x and the next token y) is produced from a set of latent variables z that correspond to interpretable semantic attributes such as topic, sentiment, tense, or syntactic role. Under this model, the authors derive a theoretical relationship between the LLM’s internal representation f(x) and the latent concepts. Assuming three mild conditions—(i) diversity of token embeddings, (ii) near‑zero conditional entropy H(z|x) so that the context almost determines the concepts, and (iii) smooth variation of concept posteriors across tokens—they prove that the representation can be expressed as a linear mixture of the log‑posteriors of each concept:

 f(x) ≈ A ·