PERSPECTRA: A Scalable and Configurable Pluralist Benchmark of Perspectives from Arguments

Pluralism, the capacity to engage with diverse perspectives without collapsing them into a single viewpoint, is critical for developing large language models that faithfully reflect human heterogeneity. Yet this characteristic has not been carefully examined in the LLM research community and remains absent from most alignment studies. Debate-oriented sources provide a natural entry point for pluralism research. Previous work builds on online debate sources but remains constrained by costly human validation. Other debate-rich platforms such as Reddit and Kialo also offer promising material: Reddit provides linguistic diversity and scale but lacks clear argumentative structure, while Kialo supplies explicit pro/con graphs but remains overly concise and detached from natural discourse. We introduce PERSPECTRA, a pluralist benchmark that integrates the structural clarity of Kialo debate graphs with the linguistic diversity of real Reddit discussions. Using a controlled retrieval-and-expansion pipeline, we construct 3,810 enriched arguments spanning 762 pro/con stances on 100 controversial topics. Each opinion is expanded to multiple naturalistic variants, enabling robust evaluation of pluralism. We initialise three tasks with PERSPECTRA: opinion counting (identifying distinct viewpoints), opinion matching (aligning supporting stances and discourse to source opinions), and polarity check (inferring aggregate stance in mixed discourse). Experiments with state-of-the-art open-source and proprietary LLMs, highlight systematic failures, such as overestimating the number of viewpoints and misclassifying concessive structures, underscoring the difficulty of pluralism-aware understanding and reasoning. By combining diversity with structure, PERSPECTRA establishes the first scalable, configurable benchmark for evaluating how well models represent, distinguish, and reason over multiple perspectives.

💡 Research Summary

The paper introduces PERSPECTRA, a novel benchmark designed to evaluate large language models’ ability to represent, distinguish, and reason over multiple viewpoints—a property the authors term “pluralism.” Existing pluralism‑oriented datasets either rely on costly human annotation, cover a narrow set of topics, or sacrifice linguistic richness for structural clarity. PERSPECTRA bridges this gap by combining the explicit pro/con graph structure of Kialo with the linguistic diversity of Reddit.

Construction proceeds in three stages. First, 100 controversial topics and their associated pro and con opinions are extracted from Kialo. Second, for each opinion a pool of semantically related Reddit comments is retrieved using Qwen3‑Embedding‑8B similarity; the top five comments per opinion are kept. Third, a controlled GPT‑4o prompting pipeline expands each (topic, opinion, comment) triple into five naturalistic argument variants that preserve the original stance while adopting Reddit‑style informality. This yields 3,810 enriched arguments covering 762 distinct opinions, each averaging about 100 words.

PERSPECTRA defines three evaluation tasks. (1) Opinion Counting asks models to infer how many distinct viewpoints are present in a mixed‑opinion paragraph, testing clustering ability under lexical variation. (2) Opinion Matching requires aligning each expanded argument back to its source Kialo opinion, measuring semantic fidelity and structural mapping. (3) Polarity Check asks models to determine the aggregate stance (pro vs. con) in a paragraph that mixes multiple arguments, probing the ability to synthesize conflicting evidence.

Experiments with seven state‑of‑the‑art models—including open‑source Llama‑2, Mistral, Falcon, and proprietary GPT‑4o and Claude—reveal systematic shortcomings. Models tend to overestimate the number of viewpoints (average F1 ≈ 0.62), misclassify concessive or nuanced structures, and struggle to correctly infer overall polarity in mixed discourse (≈ 0.68 accuracy). These failures suggest that current RLHF‑based alignment pipelines, which optimize for a single “best” answer, suppress the diversity of legitimate perspectives.

The authors acknowledge limitations: Reddit comment quality varies, embedding‑based retrieval can introduce off‑topic matches, and reliance on GPT‑4o for expansion may embed its own biases. Future work is proposed to broaden multilingual coverage, incorporate human‑LLM verification loops, and develop alignment methods that explicitly preserve pluralistic distributions.

Overall, PERSPECTRA offers a scalable, configurable resource that uniquely blends structural clarity with natural language richness, providing a rigorous testbed for pluralism‑aware language modeling and encouraging the development of more inclusive, perspective‑sensitive AI systems.