Contrastive Concept-Tree Search for LLM-Assisted Algorithm Discovery

Large language Model (LLM)-assisted algorithm discovery is an iterative, black-box optimization process over programs to approximatively solve a target task, where an LLM proposes candidate programs and an external evaluator provides task feedback. Despite intense recent research on the topic and promising results, how can the LLM internal representation of the space of possible programs be maximally exploited to improve performance is an open question. Here, we introduce Contrastive Concept-Tree Search (CCTS), which extracts a hierarchical concept representation from the generated programs and learns a contrastive concept model that guides parent selection. By reweighting parents using a likelihood-ratio score between high- and low-performing solutions, CCTS biases search toward useful concept combinations and away from misleading ones, providing guidance through an explicit concept hierarchy rather than the algorithm lineage constructed by the LLM. We show that CCTS improves search efficiency over fitness-based baselines and produces interpretable, task-specific concept trees across a benchmark of open Erdős-type combinatorics problems. Our analysis indicates that the gains are driven largely by learning which concepts to avoid. We further validate these findings in a controlled synthetic algorithm-discovery environment, which reproduces qualitatively the search dynamics observed with the LLMs.

💡 Research Summary

The paper addresses the emerging field of large‑language‑model (LLM)‑assisted algorithm discovery, where a generative LLM proposes candidate programs and an external evaluator supplies a scalar performance score. Existing methods treat the search as a black‑box evolutionary process that selects parents purely on fitness, ignoring the rich semantic structure implicitly learned by the LLM. The authors propose Contrastive Concept‑Tree Search (CCTS), a framework that extracts a hierarchical “concept tree” from each generated program, learns a contrastive model over high‑ and low‑performing programs, and uses the resulting likelihood‑ratio scores to bias parent selection toward useful concept combinations while steering away from harmful ones.

Key components of CCTS

1. Concept extraction – After each child program is generated, a lightweight prompt asks the LLM to list the algorithmic concepts present (e.g., “graph coloring”, “greedy heuristic”, “DFS”). The returned set is encoded as a binary indicator vector over a dynamically growing set of concepts V. The concept hierarchy is maintained as a rooted tree; activation is ancestor‑closed, meaning that selecting a refined concept automatically activates all its ancestors. New concepts are added to V whenever they first appear.

2. Contrastive learning – The archive Aₜ of evaluated programs is split at each iteration into a “good” set (scores ≥ τₜ) and a “bad” set (scores < τₜ). Two probability models ˆp₊(b) and ˆp₋(b) are fitted to the binary concept vectors of the good and bad subsets respectively, using a shared parametric family (e.g., a factorized Bernoulli or a simple neural density estimator). Parameters are updated by maximum‑likelihood (cross‑entropy) on the respective subsets, with smoothing and L2 regularization to avoid over‑fitting.

3. Parent weighting – For each candidate parent i with concept vector bᵢ, a likelihood‑ratio score L(bᵢ)=log ˆp₊(bᵢ)−log ˆp₋(bᵢ) is computed. The parent‑selection distribution πₜ mixes a uniform exploration component (probability p_explore) with a CCTS‑biased component (probability 1−p_explore) proportional to exp(L(bᵢ)). This mixture enables a controllable exploration‑exploitation trade‑off.

4. Search loop – The overall evolutionary loop follows the standard pattern: select parents according to πₜ, prompt the LLM (conditioned on the task description and a context built from the archive) to generate a child, evaluate the child with the external evaluator, extract its concept vector, update the archive and the concept tree, and finally update the contrastive models.

Empirical evaluation – The authors test CCTS on a suite of open Erdős‑type combinatorial problems (graph coloring, matching, routing, etc.). Baselines include pure uniform sampling, greedy fitness‑based selection, k‑elite selection, and recent LLM‑based systems such as AlphaEvolve and ThetaEvolve. Across all tasks, CCTS reaches comparable or better solutions with roughly 2.3× fewer evaluator calls. Notably, in the graph‑coloring benchmark CCTS achieves a 30 % higher success rate within the same budget.

Interpretability – Because the model operates in a human‑readable concept space, the authors can visualize the evolving concept tree. High‑performing programs tend to accumulate concepts like “greedy coloring → color swap → conflict resolution”, whereas low‑performing programs contain noisy concepts such as “random walk”. The likelihood‑ratio weighting effectively suppresses parents that contain detrimental concepts, a mechanism that traditional fitness‑only methods cannot exploit.

Synthetic validation – To isolate the effect of contrastive learning, the authors construct a synthetic algorithm‑discovery environment where concepts are artificially labeled as beneficial or harmful. CCTS quickly learns the correct contrastive scores, confirming that the method can identify useful concepts even when the underlying program space is arbitrarily complex.

Limitations and future work – The current implementation incurs extra cost for concept extraction (an additional LLM call per child). Scaling to large codebases may require a dedicated concept encoder or a cached extraction mechanism. The hierarchy construction relies on human‑defined refinement relations; automated hierarchy learning (e.g., clustering + Bayesian network inference) would make the approach more general. Finally, extending CCTS to multi‑task settings and investigating transfer of learned concept utilities across tasks are promising directions.

Conclusion – CCTS demonstrates that making the latent semantic structure of LLM‑generated programs explicit and learning a contrastive model over that structure yields substantial gains in search efficiency and provides interpretable insight into algorithmic building blocks. The work opens a pathway toward more principled, concept‑driven exploration in LLM‑assisted scientific discovery.