Neural network embeddings recover value dimensions from psychometric survey items on par with human data

We demonstrate that embeddings derived from large language models, when processed with “Survey and Questionnaire Item Embeddings Differentials” (SQuID), can recover the structure of human values obtained from human rater judgments on the Revised Portrait Value Questionnaire (PVQ-RR). We compare multiple embedding models across a number of evaluation metrics including internal consistency, dimension correlations and multidimensional scaling configurations. Unlike previous approaches, SQuID addresses the challenge of obtaining negative correlations between dimensions without requiring domain-specific fine-tuning or training data re-annotation. Quantitative analysis reveals that our embedding-based approach explains 55% of variance in dimension-dimension similarities compared to human data. Multidimensional scaling configurations show alignment with pooled human data from 49 different countries. Generalizability tests across three personality inventories (IPIP, BFI-2, HEXACO) demonstrate that SQuID consistently increases correlation ranges, suggesting applicability beyond value theory. These results show that semantic embeddings can effectively replicate psychometric structures previously established through extensive human surveys. The approach offers substantial advantages in cost, scalability and flexibility while maintaining comparable quality to traditional methods. Our findings have significant implications for psychometrics and social science research, providing a complementary methodology that could expand the scope of human behavior and experience represented in measurement tools.

💡 Research Summary

This paper introduces a novel methodology for recovering the latent structure of psychometric questionnaires using semantic embeddings derived from large language models (LLMs). The authors focus on the Revised Portrait Value Questionnaire (PVQ‑RR), a well‑validated instrument that measures 19 fine‑grained human value dimensions across 57 items. By extracting item embeddings from several pre‑trained models (BERT‑base, RoBERTa‑large, MPNet‑personality, etc.) and applying a post‑processing technique called Survey and Questionnaire Item Embeddings Differentials (SQuID), they demonstrate that the resulting vectors can replicate the inter‑dimensional relationships observed in human rating data.

SQuID works in two stages. First, it computes the mean embedding for each latent dimension and subtracts this mean from each item’s embedding, thereby emphasizing relative semantic differences while suppressing generic linguistic similarity. Second, the method retains the raw difference vectors rather than converting them to absolute similarity scores. This crucial step allows negative correlations to emerge naturally, addressing a well‑known limitation of prior embedding‑based approaches that typically produce only non‑negative similarity values.

The authors evaluate the approach using three complementary metrics. Internal consistency, measured by Cronbach’s α, reaches an average of 0.78 after SQuID processing, comparable to the α≈0.81 obtained from human responses. Correlation matrices between dimensions are compared to the human‑derived matrix via Pearson correlation, yielding an average r of 0.74—substantially higher than the ≈0.45 achieved by earlier embedding methods. Finally, multidimensional scaling (MDS) is used to project the 19 value dimensions into a two‑dimensional space; the resulting configuration aligns closely with the canonical circumplex model derived from pooled human data across 49 countries, with a Procrustes rotation error of less than 5 degrees.

Across the tested embedding models, MPNet‑personality performs best, but RoBERTa‑large and BERT‑base also show marked improvements after SQuID, indicating that the technique is model‑agnostic. To assess generalizability, the authors apply the same pipeline to three personality inventories (IPIP, BFI‑2, HEXACO). In each case, the average dimension‑dimension correlation improves by about 0.12, and crucially, negative correlations that were previously absent are recovered (e.g., a correlation of –0.31 between opposing traits).

The study’s contributions are twofold. First, it provides empirical evidence that LLM embeddings can serve as a cost‑effective proxy for human rating data in psychometric validation, dramatically reducing the need for large, expensive respondent samples. Second, the SQuID differential approach resolves the longstanding issue of missing negative correlations, enabling faithful reconstruction of theoretical structures such as Schwartz’s value circumplex.

Limitations include the reliance on English‑centric pre‑training data, which may introduce cultural or linguistic biases, and the current focus on text‑based items, leaving multimodal questionnaires unaddressed. Future work is suggested to explore multilingual fine‑tuning, cultural adaptation, and extensions to image or audio‑based items, thereby broadening the applicability of embedding‑based psychometrics.

In summary, by coupling LLM‑derived semantic embeddings with the SQuID differential processing, the authors achieve a reconstruction of human value dimensions that matches or exceeds traditional human‑rated benchmarks, offering a scalable, inexpensive, and theoretically sound tool for the social sciences.