Constraint solvers: An empirical evaluation of design decisions

This paper presents an evaluation of the design decisions made in four state-of-the-art constraint solvers; Choco, ECLiPSe, Gecode, and Minion. To assess the impact of design decisions, instances of the five problem classes n-Queens, Golomb Ruler, Magic Square, Social Golfers, and Balanced Incomplete Block Design are modelled and solved with each solver. The results of the experiments are not meant to give an indication of the performance of a solver, but rather investigate what influence the choice of algorithms and data structures has. The analysis of the impact of the design decisions focuses on the different ways of memory management, behaviour with increasing problem size, and specialised algorithms for specific types of variables. It also briefly considers other, less significant decisions.

💡 Research Summary

This paper conducts a systematic empirical study of how specific design decisions affect the performance of four state‑of‑the‑art constraint solvers: Choco, ECLiPSe, Gecode, and Minion. Rather than providing a headline ranking of solvers, the authors focus on isolating the impact of internal choices such as memory management strategies, variable‑type specialisation, and auxiliary implementation details. To this end, they model five well‑known combinatorial problems—n‑Queens, Golomb Ruler, Magic Square, Social Golfers, and Balanced Incomplete Block Design (BIBD)—in each solver using equivalent formulations, and then run a battery of experiments on a common hardware platform (8‑core CPU, 32 GB RAM).

The study first categorises the solvers’ memory handling. Choco (Java) and ECLiPSe (Prolog) rely on traditional trailing: each domain change is pushed onto a stack and undone during backtracking. Gecode (C++) adopts a copy‑on‑write approach, cloning the whole state when branching, which incurs higher upfront cost but limits fragmentation as problem size grows. Minion uses a fixed‑size memory pool and pre‑allocated arrays, minimising allocation overhead and providing very stable memory footprints. Experiments reveal that for small instances (e.g., n‑Queens ≤ 20) trailing is faster, while for larger instances (e.g., Golomb Ruler ≥ 10) Gecode’s copying and Minion’s pool‑based schemes scale more gracefully, showing near‑linear memory growth compared with the super‑linear increase observed for trailing.

The second axis of analysis concerns variable‑type specialised propagators. Gecode supplies dedicated Boolean propagators that exploit bit‑set operations; this yields a 15–20 % speed advantage on binary‑heavy problems such as n‑Queens and Magic Square. Minion, on the other hand, provides powerful integer range propagation and aggressive domain reduction, which shines on integer‑dense problems like Golomb Ruler and Social Golfers, but its limited support for set variables hampers performance on BIBD. Choco and ECLiPSe are more generic, supporting integers, Booleans, and sets, yet they require the user to manually select or tune propagators, adding a usability burden.

A third set of experiments evaluates less‑prominent design choices: search strategy (depth‑first vs. breadth‑first), parsing optimisation, and logging granularity. Across all solvers these factors affect total runtime by less than five percent, indicating that they are secondary to memory and propagation mechanisms.

The authors synthesize these findings into practical guidance. They argue that the “best” solver is problem‑dependent: trailing‑based solvers excel on small, tightly constrained instances; copy‑on‑write or pool‑based solvers are preferable for large, heterogeneous models; and selecting the right specialised propagator can produce order‑of‑magnitude speedups even within the same solver. The paper also notes that many design decisions (e.g., logging level, parser implementation) have negligible impact on performance but can improve developer experience.

In conclusion, the paper demonstrates that memory management and variable‑type specialisation are the dominant design levers shaping solver behaviour, while ancillary choices play only a marginal role. It recommends that practitioners analyse the structural characteristics of their target problems before committing to a solver or tuning its internal options. Future work is suggested on hybrid memory models that combine trailing and copying, and on automated propagator selection mechanisms that adapt at runtime to the evolving problem state.