Althea: Human-AI Collaboration for Fact-Checking and Critical Reasoning

The web’s information ecosystem demands fact-checking systems that are both scalable and epistemically trustworthy. Automated approaches offer efficiency but often lack transparency, while human verification remains slow and inconsistent. We introduce Althea, a retrieval-augmented system that integrates question generation, evidence retrieval, and structured reasoning to support user-driven evaluation of online claims. On the AVeriTeC benchmark, Althea achieves a Macro-F1 of 0.44, outperforming standard verification pipelines and improving discrimination between supported and refuted claims. We further evaluate Althea through a controlled user study and a longitudinal survey experiment (N = 642), comparing three interaction modes that vary in the degree of scaffolding: an Exploratory mode with guided reasoning, a Summary mode providing synthesized verdicts, and a Self-search mode that offers procedural guidance without algorithmic intervention. Results show that guided interaction produces the strongest immediate gains in accuracy and confidence, while self-directed search yields the most persistent improvements over time. This pattern suggests that performance gains are not driven solely by effort or exposure, but by how cognitive work is structured and internalized.

💡 Research Summary

**
The paper introduces Althea, a retrieval‑augmented fact‑checking platform that blends question generation, evidence retrieval, and structured reasoning to enable human‑AI collaboration in evaluating online claims. Recognizing the trade‑off between the scalability of automated systems and the epistemic trustworthiness of human verification, the authors design Althea to provide transparent, interactive support rather than a passive verdict. The system consists of four core modules: (1) a Source Analyzer that extracts meta‑information (source type, political bias, etc.) to establish an initial credibility frame; (2) an Expert Finder that leverages the Google Fact‑Check Tools API and a GPT‑Oss‑1‑20B model to retrieve and summarize professional fact‑checks; (3) a Perspective Integrator that uses the Perplexity Sonar API to surface opposing and supporting viewpoints, thereby exposing the argumentative landscape; and (4) an Evidence Synthesizer that classifies evidence at the sentence level (support, refute, insufficient) and aggregates claim‑level judgments.

Althea offers three interaction modes that vary in scaffolding intensity. The Exploratory mode visualizes a step‑by‑step loop of question → evidence → reasoning, giving users immediate feedback after each retrieval and encouraging iterative refinement of their judgments. The Summary mode presents a concise, AI‑generated verdict with key supporting evidence for rapid decision‑making, but it reduces user agency. The Self‑search mode provides procedural guidance without delivering a final verdict, prompting users to conduct the verification themselves while still benefiting from the system’s retrieval and summarization tools.

Evaluation on the AVeriTeC benchmark—a collection of thousands of real‑world claims annotated by professional fact‑checking organizations—shows that Althea achieves a Macro‑F1 score of 0.44, surpassing standard verification pipelines. In a controlled user study (N = 642) and a longitudinal survey (N = 642, spanning 4 weeks), the three modes produced distinct patterns. Exploratory mode yielded the highest immediate gains in accuracy (≈23 % improvement over baseline) and confidence (1.8‑point increase). Self‑search mode, while initially lagging by about 5 % in accuracy, generated the most persistent improvements: participants’ accuracy continued to rise by an additional 7 % after two weeks, and self‑efficacy scores grew by 3.2 points, indicating internalization of verification strategies. Summary mode delivered fast but less durable performance gains. Qualitative interviews highlighted that users valued the transparency of evidence traces, the ability to “see” how the system reasoned, and the scaffolding that prevented them from falling into confirmation bias.

From an engineering perspective, Althea transitioned from a local retrieval engine to the Perplexity Sonar API, cutting latency by roughly 30 % and reducing external API costs through intelligent caching. All interaction logs and evidence trails are stored encrypted in an AWS RDS instance, with strict compliance to privacy and IRB guidelines, demonstrating that collaborative fact‑checking can meet ethical and legal standards.

The authors conclude that the degree of scaffolding—how much the system guides versus lets users act independently—critically shapes both short‑term performance and long‑term reasoning skill development. Althea’s modular architecture and empirical findings provide a blueprint for future fact‑checking tools that aim to balance automation with human critical thinking. Prospective work includes domain‑specific prompt tuning, multimodal evidence integration (images, video), and large‑scale longitudinal studies to further refine human‑AI collaborative verification.