Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization

Emotion is a key element in user-generated videos. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-generated content and the sparsity of video frames expressing emotion. In this paper, for the first time, we study the problem of transferring knowledge from heterogeneous external sources, including image and textual data, to facilitate three related tasks in understanding video emotion: emotion recognition, emotion attribution and emotion-oriented summarization. Specifically, our framework (1) learns a video encoding from an auxiliary emotional image dataset in order to improve supervised video emotion recognition, and (2) transfers knowledge from an auxiliary textual corpora for zero-shot recognition of emotion classes unseen during training. The proposed technique for knowledge transfer facilitates novel applications of emotion attribution and emotion-oriented summarization. A comprehensive set of experiments on multiple datasets demonstrate the effectiveness of our framework.

💡 Research Summary

This paper presents a novel framework for understanding emotions in user-generated videos by transferring knowledge from heterogeneous external sources, namely images and text. The core challenge addressed is the difficulty of analyzing emotional content in such videos due to their unstructured nature and the sparsity of frames that actually convey emotion.

The framework tackles three interconnected tasks: video emotion recognition, emotion attribution, and emotion-oriented summarization. For emotion recognition, the authors propose a two-pronged knowledge transfer approach. First, they introduce an Image Transfer Encoding (ITE) algorithm. ITE leverages a large auxiliary dataset of emotional images (e.g., SentiBank) to learn an emotion-centric dictionary via clustering of deep CNN features. This dictionary is then used to encode a video by aggregating its frame-level features into a single, emotion-sensitive video-level representation, effectively amplifying the signal from sparse emotional frames.

Second, to enable recognition of emotion categories not seen during training (zero-shot learning), the framework transfers knowledge from large textual corpora. Using word embedding models like Word2Vec, a semantic vector space is constructed where words for emotions (e.g., “joy,” “nostalgia”) reside. A mapping function is learned to project the ITE-based video representation into this semantic space. Consequently, the system can recognize novel emotions by comparing the video’s projected vector to the word vectors of unseen emotion labels, leveraging semantic relationships learned from text (e.g., “joy” + “trust” is close to “love”).

The proposed ITE representation naturally facilitates two novel applications. Emotion Attribution involves identifying which specific frames or clips within a video contribute most to its overall emotional label, based on the attribution scores derived during the ITE encoding process. Emotion-Oriented Summarization formulates video summarization as an optimization problem that balances three criteria: maximizing the emotional content (using attribution scores), maximizing the informational diversity of the selected clips, and adhering to a desired summary length. This ensures the summary is not only concise but also emotionally resonant.

The authors conduct comprehensive experiments on multiple datasets, including two newly introduced emotion-centric video datasets, VideoStory-P14 and YF-E6. Results demonstrate that: 1) The ITE method outperforms baseline and state-of-the-art methods for supervised video emotion recognition. 2) The zero-shot recognition approach effectively identifies unseen emotion classes by utilizing semantic knowledge from text. 3) The emotion attribution method successfully highlights key emotional moments in videos. 4) User studies confirm that the emotion-oriented summaries are preferred over summaries generated by methods that do not consider emotional content.

In summary, this work makes significant contributions by being the first to explore zero-shot video emotion recognition via heterogeneous knowledge transfer, and by formally defining and providing solutions for the novel tasks of emotion attribution and emotion-oriented video summarization, offering a holistic framework for advanced video emotion understanding.