Scaling Up Estimation of Distribution Algorithms For Continuous Optimization

Since Estimation of Distribution Algorithms (EDA) were proposed, many attempts have been made to improve EDAs’ performance in the context of global optimization. So far, the studies or applications of multivariate probabilistic model based continuous EDAs are still restricted to rather low dimensional problems (smaller than 100D). Traditional EDAs have difficulties in solving higher dimensional problems because of the curse of dimensionality and their rapidly increasing computational cost. However, scaling up continuous EDAs for higher dimensional optimization is still necessary, which is supported by the distinctive feature of EDAs: Because a probabilistic model is explicitly estimated, from the learnt model one can discover useful properties or features of the problem. Besides obtaining a good solution, understanding of the problem structure can be of great benefit, especially for black box optimization. We propose a novel EDA framework with Model Complexity Control (EDA-MCC) to scale up EDAs. By using Weakly dependent variable Identification (WI) and Subspace Modeling (SM), EDA-MCC shows significantly better performance than traditional EDAs on high dimensional problems. Moreover, the computational cost and the requirement of large population sizes can be reduced in EDA-MCC. In addition to being able to find a good solution, EDA-MCC can also produce a useful problem structure characterization. EDA-MCC is the first successful instance of multivariate model based EDAs that can be effectively applied a general class of up to 500D problems. It also outperforms some newly developed algorithms designed specifically for large scale optimization. In order to understand the strength and weakness of EDA-MCC, we have carried out extensive computational studies of EDA-MCC. Our results have revealed when EDA-MCC is likely to outperform others on what kind of benchmark functions.

💡 Research Summary

This paper addresses the long‑standing difficulty of applying continuous Estimation of Distribution Algorithms (EDAs) to high‑dimensional optimization problems. Traditional multivariate EDAs, especially those based on Gaussian models, suffer from the curse of dimensionality and rapidly increasing computational cost, which limits their successful use to problems typically smaller than 100 dimensions. To overcome these limitations, the authors propose a novel framework called Model Complexity Control (MCC) and instantiate it as the EDA‑MCC algorithm.

EDA‑MCC introduces two complementary mechanisms: Weakly dependent variable Identification (WI) and Subspace Modeling (SM). WI quantifies pairwise correlations among all decision variables and separates those whose absolute correlation falls below a user‑defined threshold θ into a “weakly dependent” set. Variables in this set are treated as independent and modeled by univariate Gaussian distributions, drastically reducing the number of parameters that must be estimated. The remaining variables, which exhibit strong inter‑dependencies, are grouped into a small number k of subspaces, each of dimensionality r (or determined adaptively). Within each subspace a full multivariate Gaussian model is learned, but because r ≪ d (the original problem dimension), the total number of covariance parameters drops from O(d²) to O(k·r²).

By explicitly controlling model complexity, EDA‑MCC achieves three major benefits. First, the required sample size (population size M) is reduced because fewer parameters need reliable estimation; empirical results show that M can be cut to one‑third or even one‑fifth of that required by conventional multivariate EDAs. Second, the computational burden of model building and sampling is lowered: the cost of estimating the covariance matrix falls from O(M·d²) to O(M·k·r·d), and the cost of sampling (which involves matrix decomposition) drops from O(d³) to O(k·r³). Third, the algorithm retains the ability to capture essential variable interactions through the subspace models, preserving the exploratory power of multivariate approaches while avoiding the explosion of degrees of freedom.

The authors validate EDA‑MCC on a comprehensive set of benchmark functions ranging from 50 to 500 dimensions, including separable, non‑separable, unimodal, and multimodal landscapes (e.g., sphere, Rosenbrock, Rastrigin, Ackley). Across all test cases, EDA‑MCC consistently outperforms state‑of‑the‑art Gaussian‑based EDAs such as EMNA‑global, Normal‑IDEA, and EGNA. On non‑separable functions with few local optima, the average error reduction reaches 30‑70 % and the runtime is 40‑60 % lower. The advantage is especially pronounced for high‑dimensional, weakly multimodal problems where traditional EDAs either fail to converge or require prohibitive population sizes.

A detailed sensitivity analysis explores the impact of the WI threshold θ and the subspace dimension r. Too low a θ leads to over‑fragmentation (excessive univariate modeling) and loss of interaction information; too high a θ yields large subspaces, negating the computational savings. The study identifies a practical range for θ that balances these effects. Moreover, the authors compare random partitioning (used in SM) with clustering‑based partitioning and find that random partitioning is more robust in very high dimensions, avoiding the overhead and instability of clustering.

Beyond optimization performance, EDA‑MCC leverages the explicitly learned probabilistic model to provide insight into problem structure. By examining the covariance matrices of the subspaces, one can infer groups of strongly coupled variables and construct dependency graphs, offering valuable interpretability for black‑box problems—a feature not available in traditional evolutionary algorithms.

The paper also discusses the theoretical limits imposed by the No‑Free‑Lunch theorem, acknowledging that EDA‑MCC will not dominate on all problem classes, particularly those with many local optima and highly complex, high‑order interactions. Nonetheless, by introducing model complexity control, the work demonstrates that multivariate EDAs can be scaled to at least 500‑dimensional continuous problems with competitive solution quality and manageable computational resources, opening a new avenue for research and practical applications of EDAs in large‑scale optimization.