SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editing

Recent advancements in Generative Artificial Intelligence (GenAI) have significantly enhanced the capabilities of both image generation and editing. However, current approaches often treat these tasks separately, leading to inefficiencies and challenges in maintaining spatial consistency and semantic coherence between generated content and edits. Moreover, a major obstacle is the lack of structured control over object relationships and spatial arrangements. Scene graph-based methods, which represent objects and their interrelationships in a structured format, offer a solution by providing greater control over composition and interactions in both image generation and editing. To address this, we introduce SimGraph, a unified framework that integrates scene graph-based image generation and editing, enabling precise control over object interactions, layouts, and spatial coherence. In particular, our framework integrates token-based generation and diffusion-based editing within a single scene graph-driven model, ensuring high-quality and consistent results. Through extensive experiments, we empirically demonstrate that our approach outperforms existing state-of-the-art methods.

💡 Research Summary

SimGraph introduces a unified framework that simultaneously handles image generation and editing driven by scene graphs. The authors observe that current generative AI systems treat generation and editing as separate pipelines, which leads to inefficiencies, loss of spatial consistency, and difficulty preserving background details when edits are applied. By leveraging scene graphs—a structured representation of objects (nodes) and their relationships (edges)—SimGraph provides fine‑grained, interpretable control over composition, layout, and interactions.

The system consists of three main components: (1) scene‑graph extraction, (2) token‑based image generation, and (3) diffusion‑based image editing. For extraction, a multimodal large language model (e.g., Qwen‑VL) parses an input image to produce a set of triplets (subject, relation, object). Redundant or bidirectional relations are pruned, and the remaining triplets are ordered by a salience score that combines object size and position, yielding a compact caption C that reflects the graph’s semantic hierarchy.

In the generation path, the caption C is encoded with a frozen CLIP text encoder to obtain an embedding e_t. A Visual AutoRegressive (VAR) model, built on a Transformer, predicts a sequence of discrete visual tokens z₁…z_L conditioned on e_t. A pretrained VQ‑VAE decoder then maps these tokens back to pixel space, producing an image that respects the spatial arrangement encoded in C. Because the token order follows the salience‑sorted triplets, the generated layout aligns closely with the original graph.

For editing, the original graph G is first extracted from the source image I, and a user instruction T is used to modify the graph into G′. Algorithm 1 constructs two complementary textual prompts: a source prompt T_src that describes the unchanged (background) relations, and a target prompt T_tgt that describes the new or altered relations while also retaining the background context. These prompts are fed into a diffusion editor (LEDIT++), which first inverts the source image into latent space using DDIM inversion guided by T_src, then performs a denoising trajectory with joint conditioning on both T_src and T_tgt. The UNet predicts an unconditional branch and two conditioned branches; the branches are blended using weights w_src, w_tgt and a classifier‑free guidance scale s>1. This joint conditioning stabilizes background regions while driving the desired modifications, and the final latent is decoded back to an edited image.

Training uses a conditional negative log‑likelihood loss. For generation, the supervision is the VQ‑VAE encoder output; for editing, it is the standard diffusion noise‑prediction loss. Importantly, the same parameter set θ is shared across both tasks, eliminating the need for separate models and reducing memory and compute overhead.

Extensive experiments on COCO‑Stuff and Visual Genome derived datasets compare SimGraph against state‑of‑the‑art scene‑graph generation methods (SG2IM, SGDiff) and editing methods (SGEdit, SIMSG). Quantitative metrics—FID, Inception Score, relation accuracy, and an edit consistency score—show SimGraph consistently outperforms baselines, especially on complex graphs with many objects and relationships. Qualitative user studies confirm higher perceived realism, better background preservation, and more accurate relational changes. Moreover, because generation and editing share a single pipeline, inference time and GPU memory usage drop by roughly 30‑40 % compared to running two separate models.

The paper also discusses limitations: reliance on the accuracy of the initial scene‑graph extraction, challenges extending the approach to video (temporal consistency), and sensitivity to hyperparameters such as guidance scale and branch weights. Future work may explore more robust graph parsers, automatic hyperparameter tuning, and temporal extensions.

In summary, SimGraph demonstrates that a scene‑graph‑driven, jointly conditioned architecture can unify image synthesis and manipulation, delivering higher fidelity, spatial coherence, and computational efficiency than existing fragmented solutions. This unified approach opens avenues for interactive design tools, content creation pipelines, and any application requiring precise, structured control over visual media.