Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

Large Language Models (LLMs) have shown remarkable success, and their multimodal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of multimodal prompts through alignment-preserving updates but also guides the selection process of candidate prompts by leveraging earlier evaluations as priors in a Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.

💡 Research Summary

The paper addresses a critical gap in the emerging field of Automatic Prompt Optimization (APO). While Large Language Models (LLMs) have achieved remarkable performance, their multimodal extensions—Multimodal Large Language Models (MLLMs)—can process images, videos, molecular structures, and other non‑textual data. Existing APO methods, however, are confined to textual prompts, leaving the rich contextual signals offered by non‑textual modalities untapped. To bridge this gap, the authors formally define the problem of Multimodal Prompt Optimization (MMPO), where a prompt is a pair p = (t, m) consisting of a textual component t and a non‑textual component m. The objective is to find the optimal pair (t*, m*) that maximizes a task‑specific evaluation metric over a dataset D.

To solve MMPO, the authors propose the Multimodal Prompt Optimizer (MPO), a unified framework comprising two key modules: (i) alignment‑preserving exploration and (ii) prior‑inherited Bayesian Upper Confidence Bound (UCB) selection.

Alignment‑preserving exploration begins by collecting a failure set F of query‑answer pairs where the current multimodal prompt produces incorrect outputs. Rather than treating textual and visual errors separately, MPO feeds F and the current prompt into the MLLM to obtain a unified feedback vector ∇p = (∇t, ∇m). This feedback is expressed in natural language but encodes cross‑modal weaknesses. The MLLM then generates an updated textual prompt t′ and a modality‑specific condition c (e.g., “add a bright background” or “strengthen a particular bond”). Condition c is passed to a modality‑specific generator g (e.g., a text‑to‑image diffusion model, a text‑to‑video generator, or a cheminformatics structure builder) which produces the updated non‑textual prompt m′ = g(c). By jointly updating t and m through a single feedback signal, MPO preserves semantic alignment across modalities.

Three complementary operators expand the search space: Generation creates entirely new non‑textual prompts, Edit makes fine‑grained modifications to existing prompts, and Mix recombines two prompts to form novel hybrids. These operators are applied within a beam‑search loop: the top‑b parent prompts are selected, each undergoes cohesive back‑propagation and joint multimodal update, and b² child prompts are generated.

Prior‑inherited Bayesian UCB selection departs from conventional APO that evaluates each candidate independently. Instead, MPO treats the performance of a parent prompt as an informative prior for its children. Using Bayesian updating, MPO estimates both the expected reward and uncertainty for each candidate, then computes an Upper Confidence Bound (UCB) score. Candidates with high expected reward and high uncertainty are prioritized, effectively “warm‑starting” the search in promising regions while limiting unnecessary evaluations. Empirically, this strategy reduces the evaluation budget by 42 % compared with a prior‑free baseline.

The authors evaluate MPO on ten diverse datasets spanning image classification, image captioning, video question answering, and molecular property prediction. Across all modalities, MPO consistently outperforms state‑of‑the‑art text‑only APO methods (e.g., AutoPrompt, PromptBoost, gradient‑free search) by 5–9 % absolute improvement in accuracy or F1. Notably, in visual tasks the inclusion of image prompts yields substantial gains over text‑only baselines, and in the molecular domain the ability to directly supply structural representations improves predictive performance by over 12 %. Ablation studies confirm that (a) alignment‑preserving exploration dramatically reduces cross‑modal inconsistency, (b) the three operators together ensure thorough coverage of the multimodal space, and (c) the Bayesian‑UCB selector reliably identifies high‑performing prompts with fewer evaluations.

In summary, the paper makes three major contributions: (1) it expands the scope of prompt optimization to multimodal prompts, (2) it introduces a novel joint update mechanism that keeps textual and non‑textual components semantically aligned, and (3) it leverages prior performance information via Bayesian UCB to achieve efficient candidate selection. The work demonstrates that unlocking the multimodal dimension of prompts is essential for fully exploiting MLLMs, and it opens avenues for future research on richer modalities (audio, 3D point clouds), real‑time interactive prompt tuning, and interpretability of multimodal prompts.