A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives

Multi-modal music generation, using multiple modalities like text, images, and video alongside musical scores and audio as guidance, is an emerging research area with broad applications. This paper reviews this field, categorizing music generation systems from the perspective of modalities. The review covers modality representation, multi-modal data alignment, and their utilization to guide music generation. Current datasets and evaluation methods are also discussed. Key challenges in this area include effective multi-modal integration, large-scale comprehensive datasets, and systematic evaluation methods. Finally, an outlook on future research directions is provided, focusing on creativity, efficiency, multi-modal alignment, and evaluation.

💡 Research Summary

**
This survey paper provides a comprehensive overview of the rapidly emerging field of multi‑modal music generation, where music (either symbolic scores or raw audio) is generated under the guidance of additional modalities such as text, images, and video. The authors organize the literature into three progressive categories: single‑modal, cross‑modal, and multi‑modal generation, and they examine each category from the perspectives of modality representation, alignment, integration, datasets, and evaluation.

Single‑modal generation covers the traditional paradigm in which only internal musical information (symbolic or audio) is used as input. The paper reviews a wide range of foundational models—autoregressive Transformers, VAEs, GANs, and diffusion models—detailing how they handle sequence length, pitch, rhythm, and timbre. While these approaches have achieved impressive fidelity, they lack controllability and are limited to narrow application scenarios.

Cross‑modal generation introduces a single external modality as a conditioning signal. Representative tasks include text‑to‑music, lyric‑to‑melody, and image‑to‑music synthesis. The authors discuss how modern language models (BERT, T5, FLAN‑T5) and visual encoders (CNNs, ViTs) are coupled with music decoders through mechanisms such as cross‑attention, concatenation, or conditional latent spaces. Notable systems such as MusicLM, AudioLM, and MuseNet illustrate how large‑scale pre‑training and instruction‑following can produce coherent, style‑controlled music from textual prompts.

Multi‑modal generation is the most recent trend, where two or more external modalities are fused simultaneously to guide music creation. The survey highlights three technical pillars: (1) multi‑modal feature extraction (e.g., spatiotemporal video encoders, image ViTs, audio spectrogram encoders), (2) joint embedding spaces built via contrastive learning (CLIP‑style audio‑text, AudioCLIP, Wav2CLIP), and (3) integration strategies such as cross‑attention, joint embeddings, and learned mappings. By aligning text, visual, and auditory cues, these systems can generate music that matches a narrative, visual mood, or emotional tone, opening possibilities for film scoring, game soundtracks, and therapeutic applications.

The paper then surveys datasets that support multi‑modal research, ranging from music‑only corpora (MAESTRO, LMD) to large‑scale audio‑visual collections (AudioSet‑Music, VGGSound‑Music) and text‑audio pairs (MusicCaps). It points out challenges in data acquisition, licensing, and domain bias, emphasizing the need for open, richly annotated multi‑modal benchmarks.

Evaluation is identified as a critical bottleneck. Existing metrics include objective measures (Pitch Accuracy, Onset F‑Score, Fréchet Audio Distance, BLEU/ROUGE for text‑music alignment) and subjective listening tests (MOS, A/B preference, emotion perception surveys). However, no unified metric captures the full spectrum of multi‑modal generation—namely fidelity, controllability, emotional consistency, and creative novelty—simultaneously.

Finally, the authors enumerate key challenges and future directions: (1) developing large‑scale, cross‑modal pre‑training models that can jointly understand text, images, video, and music; (2) designing efficient compression and quantization pipelines (e.g., VQ‑VAE, RVQ, EnCodec) that preserve musical semantics while enabling real‑time generation; (3) constructing comprehensive, multi‑dimensional evaluation frameworks; (4) building interactive, human‑in‑the‑loop interfaces that allow creators to steer generation with intuitive controls; and (5) curating extensive, legally sound multi‑modal datasets to fuel research. The survey concludes that while multi‑modal music generation is still in its infancy, advances in representation learning, alignment techniques, and evaluation will likely unlock unprecedented creative and commercial applications.