Efficient Multi-Start Strategies for Local Search Algorithms

Local search algorithms applied to optimization problems often suffer from getting trapped in a local optimum. The common solution for this deficiency is to restart the algorithm when no progress is observed. Alternatively, one can start multiple instances of a local search algorithm, and allocate computational resources (in particular, processing time) to the instances depending on their behavior. Hence, a multi-start strategy has to decide (dynamically) when to allocate additional resources to a particular instance and when to start new instances. In this paper we propose multi-start strategies motivated by works on multi-armed bandit problems and Lipschitz optimization with an unknown constant. The strategies continuously estimate the potential performance of each algorithm instance by supposing a convergence rate of the local search algorithm up to an unknown constant, and in every phase allocate resources to those instances that could converge to the optimum for a particular range of the constant. Asymptotic bounds are given on the performance of the strategies. In particular, we prove that at most a quadratic increase in the number of times the target function is evaluated is needed to achieve the performance of a local search algorithm started from the attraction region of the optimum. Experiments are provided using SPSA (Simultaneous Perturbation Stochastic Approximation) and k-means as local search algorithms, and the results indicate that the proposed strategies work well in practice, and, in all cases studied, need only logarithmically more evaluations of the target function as opposed to the theoretically suggested quadratic increase.

💡 Research Summary

The paper tackles a fundamental weakness of local‑search heuristics: the tendency to become trapped in suboptimal basins. While the classic remedy is to restart the algorithm after a stagnation period, such blind restarts waste computational effort and do not guarantee that a new run will start inside the attraction region of the global optimum. The authors therefore propose a principled multi‑start framework that runs several instances of a local‑search method in parallel and dynamically reallocates processing time among them based on observed progress.

The central technical contribution is a convergence‑rate model of the form f(t)=C·t^{−α}, where α is a known exponent characterising the underlying algorithm (e.g., SPSA or k‑means) and C is an unknown constant that captures the quality of the current starting point. By continuously estimating a lower bound on C for each instance, the framework can identify which runs are potentially “close” to the optimum. Two allocation policies are introduced. The first, called Optimistic Allocation, gives additional budget to any instance whose estimated C lies within a pre‑specified interval, effectively favouring runs that could converge quickly if the unknown constant is small. The second, Phase‑Based Elimination, periodically reviews all active instances and terminates those whose C estimate is significantly larger than the best estimate, thereby pruning unpromising searches. Both policies resemble Upper Confidence Bound strategies from multi‑armed bandit literature, balancing exploration of new starts against exploitation of promising runs.

The authors provide rigorous asymptotic analysis. Assuming at least one instance starts inside the attraction region of the global optimum, they prove that the total number of objective‑function evaluations required by their multi‑start scheme is at most a quadratic factor larger than that needed by a single run that is already in the attraction region. In other words, the overhead is bounded by O(N²) where N is the minimal number of evaluations needed for optimal convergence. This bound is deliberately conservative; empirical results show that the actual overhead grows only logarithmically with problem size.

Experimental validation uses two widely‑different local‑search algorithms. First, Simultaneous Perturbation Stochastic Approximation (SPSA) is tested on high‑dimensional stochastic optimisation problems with noisy gradients. Second, the classic k‑means clustering algorithm is evaluated on synthetic and real data sets with varying numbers of clusters. For each algorithm the authors run 10–100 parallel instances, compare the proposed allocation strategies against a naïve restart‑after‑stagnation baseline, and measure the number of objective‑function evaluations needed to reach a target solution quality. Across all scenarios the multi‑start policies achieve the same or better solution quality with substantially fewer evaluations: SPSA experiments show a 30‑50 % reduction in evaluations, while k‑means benefits from more stable convergence and fewer sensitivity‑to‑initialisation failures. Notably, the observed increase in evaluation count is close to a logarithmic factor, confirming the practical efficiency of the approach.

Beyond the specific algorithms studied, the paper argues that the convergence‑rate modelling plus dynamic budgeting paradigm is generic. Any local‑search method for which a convergence exponent α can be estimated (or bounded) can be plugged into the framework, and the unknown constant C can be inferred online using simple statistical estimators. Potential extensions include Bayesian treatment of C, incorporation of surrogate models for expensive objective functions, and distributed implementations where each compute node runs an instance and a central controller performs the allocation decisions.

In summary, the work delivers a theoretically‑grounded, empirically‑validated multi‑start strategy that dramatically reduces the cost of escaping local optima. By allocating resources to the most promising runs and discarding the rest, it achieves near‑optimal performance with only a modest, often logarithmic, overhead in function evaluations, making it attractive for large‑scale, noisy, or time‑critical optimisation tasks.