Fine-Tuning Language Models to Know What They Know

Metacognition is a critical component of intelligence, specifically regarding the awareness of one’s own knowledge. While humans rely on shared internal memory for both answering questions and reporting their knowledge state, this dependency in LLMs remains underexplored. This study proposes a framework to measure metacognitive ability $d_{\rm{type2}}’$ using a dual-prompt method, followed by the introduction of Evolution Strategy for Metacognitive Alignment (ESMA) to bind a model’s internal knowledge to its explicit behaviors. ESMA demonstrates robust generalization across diverse untrained settings, indicating a enhancement in the model’s ability to reference its own knowledge. Furthermore, parameter analysis attributes these improvements to a sparse set of significant modifications.

💡 Research Summary

The paper tackles the largely unexplored problem of whether large language models (LLMs) can explicitly know and report what they know—a form of metacognition. Drawing inspiration from human psychophysics, the authors introduce a dual‑prompt protocol that pairs a Direct Question (asking for a factual answer) with a Meta Question (asking the model whether it knows the answer). By treating the model’s “Yes/No” meta‑response as a confidence judgment, they compute the type‑2 signal‑detection metric d′₍type2₎, which quantifies the separation between the model’s internal confidence distributions for correct versus incorrect answers. Higher d′₍type2₎ values indicate stronger metacognitive discrimination.

To improve this capability, the authors propose Evolution Strategy for Metacognitive Alignment (ESMA), a gradient‑free optimization that perturbs the entire parameter set with Gaussian noise to generate a population of candidate models. Each candidate is evaluated on a joint reward that combines factual correctness (C) and meta‑response alignment (A):
R(C,A)=2 if C=1∧A=1 (correct and says “Yes”),
R(C,A)=1 if (C=1∧A=0) or (C=0∧A=1) (correct but says “No”, or incorrect but says “No”),
R(C,A)=0 if C=0∧A=0 (incorrect and says “Yes”).
This reward encourages models to preserve true knowledge while honestly reporting ignorance, preventing reward‑hacking where a model could simply claim “I don’t know” to avoid penalties.

The ESMA update follows the classic evolution‑strategy rule: after standardizing fitness scores, the parent parameters are shifted by a weighted sum of the noise vectors, scaled by a learning rate α. Although the method touches all parameters, empirical analysis shows that only a sparse subset (≈5 % of the total) undergoes statistically significant changes, suggesting that metacognitive alignment is driven by a few critical weights, often located in attention heads and feed‑forward layers.

Experiments span closed‑source models (OpenAI GPT‑5.2, Gemini‑3 Flash, Claude‑Sonnet) and open‑source models (Qwen2.5, Llama‑3.2, Gemma‑3) across multiple scales (1.5 B to 7 B). Baseline open‑source models exhibit low d′₍type2₎ (0.2–0.7), while after ESMA fine‑tuning all models achieve d′₍type2₎ around 0.9–1.02, surpassing even the proprietary systems. The best result, Qwen2.5‑3B + ESMA, reaches d′₍type2₎ = 1.02. ROC analyses show AUC improvements from ~0.60 (original) to ~0.75 (ESMA) across sizes, confirming that the gains are not limited to binary outputs but also hold for continuous confidence scores.

Further evaluations test generalization across languages, question styles, and newly acquired facts (post‑training knowledge). ESMA‑enhanced models consistently retain higher metacognitive performance, indicating that the alignment is not merely memorizing the training distribution but genuinely binding internal knowledge representations to explicit self‑reports. Parameter‑sparsity analysis reveals that the most modified weights cluster in specific transformer layers, offering a roadmap for future targeted fine‑tuning.

In summary, the paper makes three key contributions: (1) a standardized, signal‑detection‑based metric (d′₍type2₎) and dual‑prompt protocol for measuring LLM metacognition; (2) ESMA, a gradient‑free evolution‑strategy method that efficiently aligns factual knowledge with honest meta‑responses; and (3) empirical evidence that metacognitive improvement is driven by a small set of parameters, opening avenues for lightweight, cost‑effective model refinement. This work lays foundational tools for building LLMs that can reliably answer “What do I know?”—a crucial step toward trustworthy, self‑aware AI systems.