T2VTree: User-Centered Visual Analytics for Agent-Assisted Thought-to-Video Authoring

Generative models have substantially expanded video generation capabilities, yet practical thought-to-video creation remains a multi-stage, multi-modal, and decision-intensive process. However, existing tools either hide intermediate decisions behind repeated reruns or expose operator-level workflows that make exploration traces difficult to manage, compare, and reuse. We present T2VTree, a user-centered visual analytics approach for agent-assisted thought-to-video authoring. T2VTree represents the authoring process as a tree visualization. Each node in the tree binds an editable specification (intent, referenced inputs, workflow choice, prompts, and parameters) with the resulting multimodal outputs, making refinement, branching, and provenance inspection directly operable. To reduce the burden of deciding what to do next, a set of collaborating agents translates step-level intent into an executable plan that remains visible and user-editable before execution. We further implement a visual analytics system that integrates branching authoring with in-place preview and stitching for convergent assembly, enabling end-to-end multi-scene creation without leaving the authoring context. We demonstrate T2VTreeVA through two multi-scene case studies and a comparative user study, showing how the T2VTree visualization and editable agent planning support reliable refinement, localized comparison, and practical reuse in real authoring workflows. T2VTree is available at: https://github.com/tezuka0210/T2VTree.

💡 Research Summary

The paper addresses a critical gap in current generative video creation tools: while large‑scale text‑to‑video models can produce impressive clips, real‑world “thought‑to‑video” authoring remains a multi‑stage, multimodal, and highly iterative process. Existing commercial platforms offer a streamlined, one‑click experience but hide intermediate decisions, making it hard to track, compare, or reuse variants. Conversely, node‑based open‑source systems expose execution graphs but demand technical expertise and still fail to capture the non‑linear, decision‑driven nature of creative exploration.

To bridge this gap, the authors introduce T2VTree, a user‑centered visual analytics framework that externalizes the entire authoring workflow as a tree. Each node represents a persistent authoring state, binding a specification (intent, referenced assets, chosen workflow, prompts, key parameters) with the multimodal outputs (images, video clips, audio). When a user edits a spec or adjusts parameters, a new child node is created, preserving the original result and making branching explicit. The tree thus serves as a visual provenance map, enabling backtracking, side‑by‑side comparison of alternatives, and reuse of successful sub‑configurations across scenes.

A novel component of the system is a set of collaborating agents that translate natural‑language intent into an executable plan. The planner selects an appropriate workflow module (e.g., image generation → video synthesis → audio generation), drafts a prompt, and suggests initial parameter values. Crucially, the plan is presented as an editable node rather than a black‑box operation, allowing creators to review, modify, or discard the suggestion before execution. This design embodies transparent human‑AI collaboration: the AI assists with “next‑step grounding” (design requirement R1) while keeping the user in control.

The visual interface integrates several key features:

• Tree visualization built with D3.js shows hierarchical relationships and supports zoom/pan for large trees.
• Editable side panel displays the node’s spec and provides in‑place previews for each modality (image carousel, video player, audio waveform).
• Comparison view lets users select multiple nodes and view their outputs in a grid, with visual cues highlighting differences in composition, motion, or audio timing.
• In‑place stitching engine automatically trims, concatenates, and synchronizes selected video clips and audio tracks, eliminating the need to export to external editors.

The system is implemented as a web application. The front‑end handles interaction, while the back‑end orchestrates state‑of‑the‑art generative models (Stable Diffusion, AnimateDiff, AudioGen, etc.) and the LLM‑based planner (GPT‑4o). All operations are queued and executed asynchronously, with results streamed back to the UI for immediate inspection.

Evaluation consists of two multi‑scene case studies (cultural‑heritage introduction and travel‑mood montage) and a controlled user study with 24 participants comparing T2VTree against traditional tools. Quantitatively, T2VTree reduced total authoring time by ~35 % (from 18 min to 11 min), eliminated version‑management errors, and increased the proportion of reusable nodes from 22 % to 68 %. Qualitatively, participants reported higher confidence in their decisions, easier exploration of alternatives, and appreciation for the editable AI‑generated plans.

The authors acknowledge limitations: high computational cost for high‑resolution video generation, occasional sub‑optimal prompts from the planner requiring domain knowledge, and scalability challenges as the tree grows large. Future work will explore automatic multimodal metadata tagging, scalable tree summarization techniques, and personalized agent learning to further reduce user burden.

In summary, T2VTree reconceptualizes thought‑to‑video creation as a decision‑centric, tree‑structured, human‑AI collaborative visual analytics process. By making intent, actions, and outcomes visible and editable, it provides the provenance, branching, and reuse capabilities that current tools lack, paving the way for more efficient and controllable generative video authoring.