Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation

The scientific ideation process often involves blending salient aspects of existing papers to create new ideas - a framework known as facet-based ideation. We contribute Scideator, the first human-LLM system for facet-based scientific ideation. Starting from a user-provided set of scientific papers, Scideator extracts key facets – purposes, mechanisms, and evaluations – from these and related papers, allowing users to explore the idea space by interactively recombining facets to synthesize inventive ideas. Scideator is driven by three design choices: (1) human-in-the-loop facet recombination, in which users select facets from retrieved papers and the system generates ideas by finding analogies across them via the Faceted Idea Generator module; (2) distance-controlled retrieval via the Analogous Paper Facet Finder module, which surfaces papers from the same topic to entirely different subareas to provide a spectrum of creative directions; and (3) facet-based novelty verification via the Idea Novelty Checker module, a retrieve-then-rerank pipeline that evaluates idea originality using facets. In a user study with computer science researchers, Scideator provided significantly more creativity support than a baseline using the same backbone LLM without our facet-based modules, particularly in idea exploration and expressiveness. Participants’ favorite ideas more often included facets selected by themselves rather than the LLM, and participants used fewer free-text instructions with Scideator, indicating a preference for facet-level steering over prompting. Finally, re-ranking papers by facet matching rather than general relevance improved novelty classification accuracy from 13.79% to 89.66%.

💡 Research Summary

Scideator is a novel human‑LLM compound system that supports scientific ideation by leveraging a structured “facet” representation of research papers. Each paper is broken down into three concise facets: purpose (the problem addressed), mechanism (the proposed solution), and evaluation (how success is measured). The system consists of three LLM‑driven modules that operate on the same facet schema, allowing users to interact at the facet level rather than with free‑form text.

Module 1 – Analogous Paper Facet Finder receives a set of seed papers (and optionally an ideation topic) and extracts purpose and mechanism facets via LLM prompts. It then generates analogical purpose‑mechanism pairs at three conceptual distances: near (same topic, different approach), far (same sub‑area, different topic), and very‑far (different sub‑area). For each distance it retrieves four candidate papers from Semantic Scholar, extracts all three facets (purpose, mechanism, evaluation) in ≤ 7‑word phrases, and builds a pool of 16 papers (plus four very‑near similarity papers). This distance‑controlled retrieval supplies a spectrum of inspiration ranging from incremental to highly divergent.

Module 2 – Faceted Idea Generator takes the pooled facets and creates ideas by recombining them across distance groups. For each cross‑distance paper pair the LLM proposes six candidate analogies, evaluates them on understandability, relevance, feasibility, specificity, and novelty, and selects the two strongest. Each selected analogy is turned into a concrete idea by swapping purpose and mechanism between the two papers (e.g., input purpose + far mechanism, or far purpose + input mechanism). Users can pre‑select a purpose, a mechanism, both, or none; the system automatically fills missing facets with complementary ones from the pool, ensuring both user‑driven direction and exploratory diversity.

Module 3 – Idea Novelty Verification assesses whether a generated idea is novel relative to existing literature. It first retrieves candidate related papers using a generic query, then re‑ranks them by facet overlap (purpose, mechanism, evaluation) rather than by generic relevance. This facet‑based re‑ranking dramatically improves novelty classification accuracy from 13.79 % (baseline) to 89.66 %. The LLM then produces a “novel / not novel” label together with a rationale that cites specific papers and explains the facet differences. When an idea is deemed non‑novel, the system suggests a single‑facet swap to increase novelty, and the user can accept the suggestion, return to the generator, or modify facet selections, forming an iterative loop.

A within‑subject user study with 22 computer‑science researchers compared Scideator against a baseline that used the same underlying LLM but omitted the facet modules. Participants performed ideation sessions with both tools. Results showed that Scideator provided significantly higher creativity support, especially in idea exploration and expressive capability. Users reported preferring facet‑level steering over free‑text prompting, and the ideas they liked most more often contained facets they had selected themselves. The novelty checker also influenced users’ self‑assessment; participants lowered their novelty ratings when the system flagged an idea as non‑novel, and they could verify the claim using the supplied related papers and explanations.

Key contributions are: (1) the first human‑LLM system that maintains a unified facet representation across retrieval, generation, and novelty assessment; (2) empirical evidence that facet‑level interaction enhances creative ideation and reduces cognitive load; (3) a facet‑based novelty verification pipeline that achieves high classification accuracy and offers actionable suggestions for improving novelty. Limitations include reliance on only three facets (purpose, mechanism, evaluation), which may be insufficient for more complex domains, and dependence on LLM prompt quality for facet extraction. Future work could expand the facet taxonomy (e.g., data, metrics, application domain), incorporate domain‑specific fine‑tuning, or employ ensemble models to increase robustness. Overall, Scideator demonstrates how structured human‑AI collaboration can systematically augment scientific creativity.