WorldEdit: Towards Open-World Image Editing with a Knowledge-Informed Benchmark

Recent advances in image editing models have demonstrated remarkable capabilities in executing explicit instructions, such as attribute manipulation, style transfer, and pose synthesis. However, these models often face challenges when dealing with implicit editing instructions, which describe the cause of a visual change without explicitly detailing the resulting outcome. These limitations arise because existing models rely on uniform editing strategies that are not equipped to handle the complex world knowledge and reasoning required for implicit instructions. To address this gap, we introduce \textbf{WorldEdit}, a dataset specifically designed to enable world-driven image editing. WorldEdit consists of high-quality editing samples, guided by paraphrased instructions that align with real-world causal logic. Furthermore, we provide \textbf{WorldEdit-Test} for evaluating the existing model’s performance on causal editing scenarios. With WorldEdit, we use a two-stage training framework for fine-tuning models like Bagel, integrating with a causal verification reward. Our results show that the proposed dataset and methods significantly narrow the gap with GPT-4o and Nano-Banana, demonstrating competitive performance not only in instruction following but also in knowledge plausibility, where many open-source systems typically struggle.

💡 Research Summary

WorldEdit: Towards Open‑World Image Editing with a Knowledge‑Informed Benchmark introduces a new dataset and evaluation suite designed to tackle a fundamental limitation of current image‑editing models: handling implicit editing instructions that describe why a visual change should occur without explicitly stating what the resulting image should look like. While recent diffusion‑based and unified multimodal models excel at explicit commands such as “remove the cat” or “apply a Van Gogh style,” they struggle when the instruction requires world knowledge and causal reasoning (e.g., “a water balloon hits a cactus”).

To fill this gap, the authors construct WorldEdit, a 11 k‑sample dataset built from publicly available segmentation datasets. The pipeline first detects objects, then pairs each object with one of ten predefined transformation types—five environment‑driven (time, temperature, humidity, acidity, light) and five mechanics‑driven (break, inflate, squeeze, twist, stretch). For each object‑type pair, a question (“What would happen if the ship broke apart?”) is generated and answered by GPT‑4o, producing a detailed textual description of the expected visual change. These descriptions are paraphrased into concise editing commands, and GPT‑4o is again used to synthesize edited images. A two‑stage filtering process discards instructions with ambiguous causality and images that lack visual plausibility, ensuring high‑quality, causally consistent triples (instruction, explanation, image).

The paper also proposes a two‑stage training framework for the open‑source unified model Bagel. Stage 1 performs supervised fine‑tuning on the paraphrased instructions, teaching the model to translate implicit commands into explicit visual transformations. Stage 2 applies reinforcement learning with a composite reward that explicitly measures (i) reasoning quality (alignment with LLM‑generated causal explanations), (ii) visual fidelity (LPIPS, CLIP‑Score), and (iii) causal consistency (adherence to physical or chemical rules). This “causal verification reward” pushes the model beyond simple text‑to‑image mapping toward world‑knowledge‑grounded editing.

Evaluation on WorldEdit‑Test, a held‑out benchmark containing 2 k implicit instructions, compares the proposed method against several baselines: diffusion‑based editors (Stable Diffusion, LDM), the AnyEdit‑style dataset models, and commercial systems (GPT‑4o, Nano‑Banana). Metrics include visual consistency, image quality, instruction following, and a newly introduced knowledge plausibility score that quantifies how well the edited output respects the underlying causal logic. The two‑stage fine‑tuned Bagel achieves state‑of‑the‑art performance among open‑source models, narrowing the gap with GPT‑4o and Nano‑Banana, especially excelling in knowledge plausibility (≈91 % accuracy).

Beyond the empirical gains, the authors discuss broader implications. WorldEdit demonstrates that large‑scale, high‑quality training data explicitly encoding causal transformations are essential for moving image‑editing research toward “open‑world” capabilities. The automated pipeline, while reliant on LLMs and generative models, offers a scalable route to expand the dataset with additional physics‑based scenarios. Future work may integrate physics simulators for even more realistic edits, explore automatic paraphrasing of implicit instructions, and apply the framework to downstream tasks such as robotic manipulation planning or virtual environment generation.

In summary, WorldEdit provides the first comprehensive benchmark that couples world knowledge, causal reasoning, and high‑fidelity image editing. By coupling a carefully curated dataset with a two‑stage training and causal‑verification reward, the paper shows that open‑source models can approach the performance of leading commercial systems on truly open‑world editing tasks, marking a significant step toward truly intelligent, knowledge‑aware visual generation.