Ctrl&Shift: High-Quality Geometry-Aware Object Manipulation in Visual Generation

Object-level manipulation, relocating or reorienting objects in images or videos while preserving scene realism, is central to film post-production, AR, and creative editing. Yet existing methods struggle to jointly achieve three core goals: background preservation, geometric consistency under viewpoint shifts, and user-controllable transformations. Geometry-based approaches offer precise control but require explicit 3D reconstruction and generalize poorly; diffusion-based methods generalize better but lack fine-grained geometric control. We present Ctrl&Shift, an end-to-end diffusion framework to achieve geometry-consistent object manipulation without explicit 3D representations. Our key insight is to decompose manipulation into two stages, object removal and reference-guided inpainting under explicit camera pose control, and encode both within a unified diffusion process. To enable precise, disentangled control, we design a multi-task, multi-stage training strategy that separates background, identity, and pose signals across tasks. To improve generalization, we introduce a scalable real-world dataset construction pipeline that generates paired image and video samples with estimated relative camera poses. Extensive experiments demonstrate that Ctrl&Shift achieves state-of-the-art results in fidelity, viewpoint consistency, and controllability. To our knowledge, this is the first framework to unify fine-grained geometric control and real-world generalization for object manipulation, without relying on any explicit 3D modeling.

💡 Research Summary

Ctrl&Shift tackles the long‑standing challenge of object‑level manipulation—relocating or rotating an object in a single image or video while preserving the surrounding scene—by unifying geometric precision and diffusion‑based generalization without requiring explicit 3D reconstruction at inference time. The authors observe that geometry‑based pipelines (e.g., NeRF, multi‑view optimization) provide fine‑grained control but suffer from poor scalability and limited real‑world applicability, whereas diffusion‑based editors excel at handling diverse, in‑the‑wild content but lack accurate pose control, leading to warped objects and inconsistent viewpoints.

The key insight of the paper is to decompose manipulation into two conceptually simple sub‑tasks: (1) object removal and (2) reference‑guided inpainting under an explicit relative camera pose. Both sub‑tasks are embedded in a single diffusion model, allowing the network to learn a unified mapping from a set of conditioning signals to the desired output. The conditioning includes (i) source video frames (background context), (ii) a reference object image that disambiguates identity, (iii) binary source and target masks, and (iv) an 8‑dimensional relative pose descriptor (axis‑angle rotation, translation, and NDC shifts).

Architecturally, Ctrl&Shift builds on a ControlNet‑style DiT (Diffusion Transformer) with a hidden size of 1536. Source frames and the reference image are encoded by a VAE encoder, while masks are down‑sampled via a space‑to‑depth (pixel‑unshuffle) operation that aligns with the VAE stride, preserving their binary nature. The relative pose vector is first Fourier‑encoded, then passed through a three‑layer MLP to produce high‑dimensional tokens that are injected via cross‑attention into the diffusion backbone. This design enables the model to receive “how the camera moves” directly, rather than learning an implicit 3D representation.

Training proceeds in a multi‑task, multi‑stage fashion. Three tasks are defined: (a) object removal (learn to inpaint the background where the object was), (b) reference‑conditioned inpainting with camera control (learn to place the object at a new pose), and (c) full manipulation (combine a and b). Each task uses dedicated loss terms (mask consistency, reconstruction fidelity, pose consistency) to keep the signals disentangled. Stage 1 focuses on learning object priors and pose control on a large synthetic‑plus‑real dataset; Stage 2 fine‑tunes on high‑quality real images to improve background preservation.

A major contribution is the scalable data‑construction pipeline. Starting from real images or video clips, an image‑to‑mesh model reconstructs a 3‑D mesh of the target object, and differentiable rasterization estimates the source camera pose. A target pose is sampled, the mesh is rendered from that viewpoint, and a pretrained reference‑inpainting model pastes the rendered object into the original background, producing paired (source, target) samples with known relative pose. The pipeline works for both still images and video sequences, providing abundant training data with realistic lighting, textures, and occlusions.

Extensive experiments are conducted on the newly introduced GeoEditBench and several existing benchmarks. Quantitatively, Ctrl&Shift outperforms prior geometry‑aware methods (e.g., GeoDiffuser, OBJect‑3DIT) and diffusion‑only editors (e.g., DragAnything, ControlNet) in PSNR, SSIM, LPIPS, and a custom viewpoint‑consistency metric. Qualitatively, user studies report higher scores for controllability, realism, and temporal coherence. Ablation studies confirm the importance of the multi‑task design, the explicit pose token, and the mask‑preserving encoding.

Limitations include reduced accuracy on highly reflective or transparent objects where pose estimation is noisy, and the current focus on single‑object manipulation. Future work may extend the framework to multi‑object scenes, integrate more sophisticated pose estimation, and develop interactive UI components (e.g., 3‑D widgets, AR gestures) for end‑users.

In summary, Ctrl&Shift introduces a novel paradigm that injects relative camera‑pose control directly into a diffusion process, achieving geometry‑consistent object manipulation without any explicit 3‑D modeling at inference. By combining a carefully engineered conditioning scheme, a multi‑task training regimen, and a realistic data generation pipeline, the method bridges the gap between precise geometric editing and the broad applicability of diffusion models, opening new possibilities for film post‑production, augmented reality, and creative visual editing.