From Vicious to Virtuous Cycles: Synergistic Representation Learning for Unsupervised Video Object-Centric Learning

Unsupervised object-centric learning models, particularly slot-based architectures, have shown great promise in decomposing complex scenes. However, their reliance on reconstruction-based training creates a fundamental conflict between the sharp, high-frequency attention maps of the encoder and the spatially consistent but blurry reconstruction maps of the decoder. We identify that this discrepancy gives rise to a vicious cycle: the noisy feature map from the encoder forces the decoder to average over possibilities and produce even blurrier outputs, while the gradient computed from blurry reconstruction maps lacks high-frequency details necessary to supervise encoder features. To break this cycle, we introduce Synergistic Representation Learning (SRL) that establishes a virtuous cycle where the encoder and decoder mutually refine one another. SRL leverages the encoder’s sharpness to deblur the semantic boundary within the decoder output, while exploiting the decoder’s spatial consistency to denoise the encoder’s features. This mutual refinement process is stabilized by a warm-up phase with a slot regularization objective that initially allocates distinct entities per slot. By bridging the representational gap between the encoder and decoder, SRL achieves state-of-the-art results on video object-centric learning benchmarks. Codes are available at https://github.com/hynnsk/SRL.

💡 Research Summary

The paper tackles a fundamental problem in unsupervised object‑centric video models that use slot‑based architectures: the encoder produces sharp, high‑frequency attention maps while the decoder, trained with a pixel‑wise reconstruction loss, yields spatially smooth but blurry reconstructions. This mismatch creates a “vicious cycle.” Noisy encoder features force the decoder to average over many possible reconstructions, leading to blurry outputs; the gradients back‑propagated from these low‑frequency reconstructions lack the fine details needed to refine the encoder’s attention, perpetuating the problem.

To break this cycle, the authors propose Synergistic Representation Learning (SRL). SRL consists of three stages: (1) a warm‑up phase with a slot‑regularization loss that prevents slot collapse and encourages each slot to specialize on a distinct object; (2) a “deblurring” path where the encoder’s sharp attention guides the decoder via a ternary contrastive loss (L_CL‑dec); (3) a “denoising” path where the decoder’s spatially coherent masks guide the encoder through a second ternary contrastive loss (L_CL‑enc).

The ternary contrastive formulation is the core technical novelty. For each anchor patch, the universal set of patches is split into (a) a positive set containing only the anchor itself, (b) a semi‑positive set comprising patches that share the same slot according to the encoder (or decoder) but have lower confidence, and (c) a negative set consisting of patches assigned to different slots. By projecting both encoder and decoder features into a shared embedding space and applying an InfoNCE‑style loss over these three groups, the model forces the decoder to sharpen object boundaries (deblurring) while simultaneously encouraging the encoder to suppress spurious long‑range associations (denoising).

The slot‑regularization loss during warm‑up computes a KL divergence between slot distributions and penalizes redundancy, effectively resetting collapsed slots early in training. Once the slots are well‑initialized, the two contrastive objectives are activated and the encoder‑decoder pair iteratively refines each other, establishing a “virtuous cycle.”

Experiments on several video object‑centric benchmarks—including CLEVR‑VID, MOVi‑A, and physics‑based simulation datasets—show that SRL outperforms prior methods such as Slot Attention, STEVE, and SlotContrast across reconstruction quality (PSNR), segmentation accuracy (ARI, mIoU), and boundary sharpness. Ablation studies confirm that (i) removing the warm‑up leads to unstable training and slot collapse, and (ii) replacing the ternary contrastive loss with a binary one dramatically reduces the deblurring effect. Visualizations illustrate markedly cleaner object masks and more precise temporal consistency.

In summary, the paper identifies the encoder‑decoder representational gap as the root cause of sub‑optimal learning in unsupervised video object‑centric models, and introduces a principled, contrastive‑based synergy that lets each component compensate for the other’s weakness. By jointly deblurring the decoder and denoising the encoder, SRL achieves state‑of‑the‑art performance and provides a compelling blueprint for future work on more complex, real‑world video understanding tasks.