AI-Assisted Scientific Assessment: A Case Study on Climate Change

The emerging paradigm of AI co-scientists focuses on tasks characterized by repeatable verification, where agents explore search spaces in ‘guess and check’ loops. This paradigm does not extend to problems where repeated evaluation is impossible and ground truth is established by the consensus synthesis of theory and existing evidence. We evaluate a Gemini-based AI environment designed to support collaborative scientific assessment, integrated into a standard scientific workflow. In collaboration with a diverse group of 13 scientists working in the field of climate science, we tested the system on a complex topic: the stability of the Atlantic Meridional Overturning Circulation (AMOC). Our results show that AI can accelerate the scientific workflow. The group produced a comprehensive synthesis of 79 papers through 104 revision cycles in just over 46 person-hours. AI contribution was significant: most AI-generated content was retained in the report. AI also helped maintain logical consistency and presentation quality. However, expert additions were crucial to ensure its acceptability: less than half of the report was produced by AI. Furthermore, substantial oversight was required to expand and elevate the content to rigorous scientific standards.

💡 Research Summary

This paper presents a case study on integrating a Gemini‑based AI assistant into a realistic scientific assessment workflow, focusing on the stability of the Atlantic Meridional Overturning Circulation (AMOC). Thirteen climate scientists from diverse sub‑disciplines collaborated with the AI system over a five‑week period, producing a comprehensive synthesis of 79 peer‑reviewed papers in a final report of roughly 8,000 words. The workflow was divided into three phases: (1) outline generation, (2) section drafting, and (3) full‑report integration. In the first phase, the AI generated a ten‑page outline which the scientists edited and used to assign responsibilities. In the second phase, after switching to the newer Gemini 3 Pro model, the AI expanded each outline item into full sections, automatically retrieving relevant literature from a pre‑curated corpus of 1,660 AMOC papers and OpenAlex, ranking sources with a hybrid BM25/dense‑retrieval approach, and inserting in‑text citations. The third phase involved iterative refinement of the whole document, with the AI checking logical flow, citation consistency, and stylistic coherence. Across all phases, 104 revision cycles were logged, and the total person‑time recorded was 46 hours 36 minutes.

Key quantitative findings include: (i) more than 90 % of AI‑generated text survived the final edit, indicating high utility for drafting and consistency; (ii) AI contributed roughly 25 % of the final content, while human experts authored about 58 % (the remainder being collaborative edits). Scientists reported perceived efficiency gains ranging from 3‑ to 17‑fold, depending on the task. The AI excelled at mechanical tasks such as summarizing literature, maintaining a unified voice, and flagging inconsistencies, but struggled with synthesizing quantitative data, identifying gaps in reasoning, and avoiding occasional hallucinations (e.g., spurious reference numbers or unfounded causal links). These shortcomings were mitigated through continuous human oversight; expert edits were essential for elevating the draft from a high‑quality overview to a rigorous scientific assessment suitable for peer review.

The authors discuss broader implications for “verification‑poor” domains—fields where ground truth cannot be experimentally confirmed and scientific consensus is built through argumentation and evidence weighting. They argue that a full‑stack AI co‑scientist must be embedded within the collaborative assessment process rather than acting as an autonomous oracle. By providing a transparent revision‑review module that evaluates argumentative structure (drawing on Toulmin’s model and rhetorical theory), the assistant supports both the presentation and logical integrity of the report. Nevertheless, the study underscores that final expert verification remains indispensable; AI currently lacks deep domain understanding and holistic reasoning required for complex climate topics like AMOC.

In conclusion, the experiment demonstrates that AI can dramatically accelerate the drafting and iterative refinement stages of climate‑science assessments, freeing researchers to focus on higher‑level synthesis and critical evaluation. However, achieving peer‑review‑level quality still depends on substantial human contribution. Future work should aim to enhance AI’s capacity for quantitative reasoning, uncertainty quantification, and more nuanced evidence integration, thereby moving toward truly synergistic human‑AI scientific collaboration.