Few-Shot Unsupervised Image-to-Image Translation

Unsupervised image-to-image translation methods learn to map images in a given class to an analogous image in a different class, drawing on unstructured (non-registered) datasets of images. While remarkably successful, current methods require access to many images in both source and destination classes at training time. We argue this greatly limits their use. Drawing inspiration from the human capability of picking up the essence of a novel object from a small number of examples and generalizing from there, we seek a few-shot, unsupervised image-to-image translation algorithm that works on previously unseen target classes that are specified, at test time, only by a few example images. Our model achieves this few-shot generation capability by coupling an adversarial training scheme with a novel network design. Through extensive experimental validation and comparisons to several baseline methods on benchmark datasets, we verify the effectiveness of the proposed framework. Our implementation and datasets are available at https://github.com/NVlabs/FUNIT .

💡 Research Summary

This paper, titled “Few-Shot Unsupervised Image-to-Image Translation,” addresses a significant limitation of existing unsupervised image-to-image translation models: their requirement for large datasets from both source and target domains during training. Inspired by the human ability to grasp the essence of a novel object from just a few examples and generalize from there, the authors propose a novel framework called FUNIT (Few-shot UNsupervised Image-to-image Translation). FUNIT aims to perform image translation to previously unseen target classes specified at test time by only a handful of example images (e.g., K=1,5,10).

The core of the FUNIT framework is a conditional generator G that takes two inputs: a content image from a source class and a set of K images from a target class. The generator is architecturally decomposed into a content encoder, a class encoder, and a decoder. The content encoder extracts a spatial content latent code representing class-invariant structural information (e.g., pose, shape) from the content image. The class encoder processes each of the K target class images individually and then computes their mean to produce a class latent code encapsulating the appearance-specific characteristics of the target class. The decoder, built with Adaptive Instance Normalization (AdaIN) residual blocks, synthesizes the output image by integrating the class code (which provides global affine transformation parameters for AdaIN layers) with the content feature map. This design enforces a separation of content (structure) and style (appearance), allowing the model to recombine them creatively for unseen classes.

Training employs a multi-task adversarial discriminator D that performs binary real/fake classification for each of the source classes seen during training. The model is optimized using a combination of three losses: 1) an adversarial (GAN) loss to ensure output images are realistic and belong to the target class, 2) a content reconstruction loss that encourages the generator to reconstruct the input image when it is used as both content and class input, promoting meaningful content representation, and 3) a feature matching loss that stabilizes training by matching intermediate feature statistics of generated images to those of real target class images.

Extensive experiments were conducted on benchmark datasets including Animal Faces and North American Birds. The evaluation compared FUNIT against strong baselines created by adapting state-of-the-art unsupervised translation models (CycleGAN, UNIT, MUNIT, StarGAN) to the few-shot setting in both “fair” (identical few-shot constraints) and “unfair” (access to all target class data during training) manners. Quantitative metrics such as human perceptual study (Top-1/Top-5 classification accuracy), Inception Score (IS), and Fréchet Inception Distance (FID) were used. Results consistently showed that FUNIT significantly outperformed all fair baselines and was competitive with or even superior to many unfair baselines, especially as the number of shot K increased. The paper also provides empirical evidence that the model’s few-shot generalization capability improves with the number of source classes available during training, mirroring the effect of broader visual experience.

Furthermore, the authors demonstrated a practical application of FUNIT for few-shot image classification. By training a classifier on images generated by FUNIT for novel classes, they achieved superior performance compared to a state-of-the-art feature hallucination method, highlighting the utility of the framework beyond pure image synthesis.

In summary, FUNIT presents a groundbreaking meta-learning approach to unsupervised image translation that achieves remarkable data efficiency and generalization to novel classes, bridging a gap towards more human-like learning and imagination in machine vision systems.