Coherent and Multi-modality Image Inpainting via Latent Space Optimization

With the advancements in denoising diffusion probabilistic models (DDPMs), image inpainting has significantly evolved from merely filling information based on nearby regions to generating content conditioned on various prompts such as text, exemplar images, and sketches. However, existing methods, such as model fine-tuning and simple concatenation of latent vectors, often result in generation failures due to overfitting and inconsistency between the inpainted region and the background. In this paper, we argue that the current large diffusion models are sufficiently powerful to generate realistic images without further tuning. Hence, we introduce PILOT (in\textbf{P}ainting v\textbf{I}a \textbf{L}atent \textbf{O}p\textbf{T}imization), an optimization approach grounded on a novel \textit{semantic centralization} and \textit{background preservation loss}. Our method searches latent spaces capable of generating inpainted regions that exhibit high fidelity to user-provided prompts while maintaining coherence with the background. Furthermore, we propose a strategy to balance optimization expense and image quality, significantly enhancing generation efficiency. Our method seamlessly integrates with any pre-trained model, including ControlNet and DreamBooth, making it suitable for deployment in multi-modal editing tools. Our qualitative and quantitative evaluations demonstrate that PILOT outperforms existing approaches by generating more coherent, diverse, and faithful inpainted regions in response to provided prompts.

💡 Research Summary

The paper introduces PILOT (Painting via Latent Optimization), a novel framework for image inpainting that leverages the power of large pre‑trained diffusion models (e.g., Stable Diffusion) without any additional fine‑tuning. Instead of relying on model retraining or simple latent/pixel blending, PILOT directly optimizes the latent vector of the masked region during the reverse diffusion process. This optimization is guided by two specially designed loss functions:

1. Background Preservation Loss (L_bg) – enforces that the reconstructed background (outside the mask) remains identical to the original image, thereby preventing unwanted changes to the unmasked area.
2. Semantic Centralization Loss (L_s) – uses cross‑attention maps to concentrate the influence of textual (or other modality) prompts onto the masked region, reducing semantic drift and ensuring that the generated content faithfully follows the user’s instructions.

The authors observe that early diffusion steps primarily encode semantic layout, while later steps add fine details. Accordingly, they introduce a coherence scale parameter γ that determines how far into the diffusion timeline the optimization runs (up to (1‑γ)·T). A larger γ yields higher fidelity at the cost of more computation; a smaller γ prioritizes speed. Additionally, an interval τ specifies how often gradients are applied (e.g., every τ steps), further balancing efficiency and quality.

The overall pipeline consists of:

• Encoding the input image into latent space (z_in) and down‑sampling the binary mask (m_d).
• Running the diffusion U‑Net with the current latent z_t and the conditioning prompts (text, reference image, sketch, etc.).
• Computing a one‑step reconstruction ˜z_0 and cross‑attention maps A_i for each attention layer.
• Updating z_t using the combined loss L = L_bg + λ·L_s (λ is a weighting factor) every τ steps until the designated γ threshold.
• After optimization, performing a standard latent blending step to smoothly merge the edited region with the untouched background until the diffusion process completes.

PILOT is model‑agnostic: it can be attached to any pre‑trained latent diffusion model, including ControlNet, DreamBooth, LoRA, and other adapters. This makes it suitable for multi‑modal editing tools where users may provide text, exemplar images, sketches, or a combination thereof. Notably, when combined with DreamBooth‑personalized models, PILOT can perform subject‑driven inpainting, preserving a specific person or object while altering only the masked area.

Experimental Validation

The authors evaluate PILOT on several benchmarks, including the PIE inpainting benchmark, NIMA, CLIP‑Score, and human preference studies. Results show:

• Quantitative superiority: PILOT outperforms state‑of‑the‑art methods such as Blended Diffusion, PFB‑Diff, and other fine‑tuned inpainting models across all metrics, with especially large gains in semantic consistency and background fidelity.
• Human preference: In a large‑scale user study (>1,000 participants), images generated by PILOT were preferred in ~68% of pairwise comparisons against competing approaches.
• Ablation studies: Removing L_bg leads to noticeable background distortion; removing L_s causes semantic drift where the generated content no longer matches the prompt. Varying γ and τ demonstrates a clear trade‑off: γ≈0.7 and τ=4 achieve a sweet spot of ~10 seconds per image on a single GPU while maintaining high visual quality.

Limitations and Future Work

While PILOT achieves impressive speed (≈10 s for 512×512 images on a single GPU), scaling to higher resolutions increases memory consumption and runtime. The reliance on cross‑attention maps means that extremely complex prompts can produce unstable gradients, occasionally degrading quality. Future research directions include multi‑scale optimization, adaptive loss weighting, and integrating self‑supervised regularizers to further improve stability and enable real‑time high‑resolution inpainting.

Conclusion

PILOT presents a new paradigm for diffusion‑based inpainting: instead of modifying the model or simply blending latents, it optimizes the latent representation itself under carefully crafted constraints that preserve background integrity and enforce prompt fidelity. By doing so, it unlocks high‑quality, multi‑modal, and computationally efficient inpainting using off‑the‑shelf diffusion models, paving the way for more versatile and user‑friendly image editing applications.