Discount Model Search for Quality Diversity Optimization in High-Dimensional Measure Spaces

Quality diversity (QD) optimization searches for a collection of solutions that optimize an objective while attaining diverse outputs of a user-specified, vector-valued measure function. Contemporary QD algorithms are typically limited to low-dimensional measures because high-dimensional measures are prone to distortion, where many solutions found by the QD algorithm map to similar measures. For example, the state-of-the-art CMA-MAE algorithm guides measure space exploration with a histogram in measure space that records so-called discount values. However, CMA-MAE stagnates in domains with high-dimensional measure spaces because solutions with similar measures fall into the same histogram cell and hence receive the same discount value. To address these limitations, we propose Discount Model Search (DMS), which guides exploration with a model that provides a smooth, continuous representation of discount values. In high-dimensional measure spaces, this model enables DMS to distinguish between solutions with similar measures and thus continue exploration. We show that DMS facilitates new capabilities for QD algorithms by introducing two new domains where the measure space is the high-dimensional space of images, which enables users to specify their desired measures by providing a dataset of images rather than hand-designing the measure function. Results in these domains and on high-dimensional benchmarks show that DMS outperforms CMA-MAE and other existing black-box QD algorithms.

💡 Research Summary

**
Quality Diversity (QD) optimization seeks a collection of solutions that simultaneously maximize a scalar objective and diversify across a user‑defined, vector‑valued measure space. Existing QD methods, such as MAP‑Elites and its recent extensions (CMA‑ME, CMA‑MAE), discretize the measure space into cells and store a scalar “discount” value per cell. The discount is used to compute an improvement signal Δ = f(θ) − f_A(m(θ)), which guides the underlying optimizer (often CMA‑ES) toward regions that promise the greatest archive improvement. This approach works well when the measure space is low‑dimensional (typically fewer than ten dimensions).

When the measure space becomes high‑dimensional, two fundamental problems arise. First, the volume of each cell grows exponentially, causing many distinct solutions to fall into the same cell. This “distortion” means that different solutions receive identical discount values, making Δ indistinguishable and effectively starving the optimizer of useful gradient information. Second, maintaining a fine‑grained histogram in high dimensions is memory‑prohibitive because the number of cells required grows exponentially with dimensionality. Prior work has attempted to mitigate distortion by increasing the number of cells, but this quickly becomes infeasible.

The authors propose Discount Model Search (DMS) to replace the discrete histogram with a smooth, continuous discount model. DMS learns a parametric function g_φ(m) (implemented as a neural network) that predicts the discount value for any measure vector m. The model is trained online using the current archive: each archived solution provides a pair (m_i, v_i) where v_i is the historical discount associated with that cell. A simple mean‑squared error loss is minimized at each iteration, updating φ with a small batch of recent archive entries. Because g_φ is continuous, even solutions with very similar measures receive distinct discount predictions, preserving a meaningful Δ signal throughout the search.

The continuous discount model integrates seamlessly with the CMA‑ES component of CMA‑MAE. The improvement values Δ computed with g_φ replace the histogram‑based Δ, and the CMA‑ES distribution parameters (mean and covariance) are updated using the same rank‑based scheme as before. This yields a QD algorithm that retains the strong objective‑driven search of CMA‑MAE while gaining robustness to high‑dimensional distortion.

A second major contribution is the introduction of Quality Diversity with Datasets of Measures (QDDM). Instead of hand‑crafting a low‑dimensional measure function, the authors treat a high‑dimensional dataset (e.g., a collection of images) as the measure space itself. Users specify desired measures simply by providing example images. In the “LSI (Hiker)” domain, the objective is to generate a photo of a hiker’s face, while the measure space consists of landscape images (beach, mountain, snow, etc.). DMS learns to map each landscape to a discount value, thereby guiding the generation of hiker images that match the visual context of each landscape. This paradigm eliminates the need for manual measure design and opens QD to any domain where large, labeled datasets exist.

Experimental evaluation covers three parts. First, standard QD benchmarks with low‑ (2‑D) and high‑dimensional (10‑D) measure spaces (LP‑Sphere) demonstrate that DMS maintains a higher number of unique archive cells over time compared to CMA‑MAE, which quickly collapses into a few cells due to distortion. DMS also achieves superior final archive quality (higher mean objective) and diversity (greater coverage of the measure space). Second, two novel image‑based QDDM domains are introduced. In both, DMS outperforms CMA‑MAE, Density Descent Search (DDS), and MAP‑Elites in terms of objective performance and the visual relevance of generated images to the target dataset. Qualitative results show that DMS produces hiker images appropriately dressed for each landscape, whereas baselines either ignore the landscape or produce low‑quality faces. Third, an ablation study examines the impact of model architecture, learning rate, and update frequency on performance, confirming that even a modest multilayer perceptron suffices and that frequent online updates are critical for tracking the evolving archive.

The paper’s contributions can be summarized as follows:

1. A continuous discount model that replaces histogram‑based discounts, eliminating distortion in high‑dimensional measure spaces and enabling sustained exploration.
2. The QDDM framework, which leverages existing datasets as high‑dimensional measures, removing the burden of hand‑crafted measure functions.
3. Empirical evidence across synthetic benchmarks and realistic image‑generation tasks showing that DMS consistently outperforms state‑of‑the‑art black‑box QD algorithms.

Overall, DMS advances the scalability of QD optimization, both in terms of memory efficiency and the ability to handle rich, high‑dimensional measures. It opens new avenues for applying QD to domains such as robot behavior repertoires, generative art, and language model red‑teamings, where the notion of “diversity” is naturally expressed in high‑dimensional feature spaces. Future work may explore richer discount model families (e.g., Gaussian processes), multi‑objective extensions, and real‑time QDDM scenarios where the measure dataset evolves during optimization.