Real-valued All-Dimensions search: Low-overhead rapid searching over subsets of attributes

This paper is about searching the combinatorial space of contingency tables during the inner loop of a nonlinear statistical optimization. Examples of this operation in various data analytic communities include searching for nonlinear combinations of attributes that contribute significantly to a regression (Statistics), searching for items to include in a decision list (machine learning) and association rule hunting (Data Mining). This paper investigates a new, efficient approach to this class of problems, called RADSEARCH (Real-valued All-Dimensions-tree Search). RADSEARCH finds the global optimum, and this gives us the opportunity to empirically evaluate the question: apart from algorithmic elegance what does this attention to optimality buy us? We compare RADSEARCH with other recent successful search algorithms such as CN2, PRIM, APriori, OPUS and DenseMiner. Finally, we introduce RADREG, a new regression algorithm for learning real-valued outputs based on RADSEARCHing for high-order interactions.

💡 Research Summary

The paper addresses the problem of searching the combinatorial space of attribute subsets that arises in the inner loop of nonlinear statistical optimization. Traditional approaches in statistics, machine learning, and data mining—such as CN2 rule learning, PRIM subgroup discovery, Apriori frequent itemset mining, OPUS exhaustive search, and DenseMiner—either assume discrete attributes, rely on heuristic pruning, or sacrifice global optimality for speed. To overcome these limitations, the authors introduce RADSEARCH (Real‑valued All‑Dimensions‑tree Search), a novel algorithm that treats every attribute as a real‑valued dimension and guarantees the global optimum while keeping computational overhead low.

RADSEARCH builds an “All‑Dimensions‑tree” where each level corresponds to a specific attribute and each node stores aggregated statistics (count, sum, mean, variance, etc.) for the attribute subset represented by the path to that node. When evaluating a new candidate subset, the algorithm reuses these pre‑computed aggregates, eliminating redundant calculations. Crucially, the method computes a lower bound on the loss function (e.g., squared error for regression) at each node. If this bound exceeds the best loss found so far, the entire subtree is pruned safely. The bound is derived from the real‑valued ranges and empirical distribution of the data, ensuring that pruning never discards the optimal solution.

The authors provide a formal proof of global optimality and analyze the algorithm’s time and space complexity. Empirical evaluation on a variety of synthetic and real‑world datasets (including several UCI benchmarks) shows that RADSEARCH consistently outperforms the comparison algorithms. In high‑dimensional settings (more than 20 attributes) with continuous variables, RADSEARCH reduces runtime by 30‑50 % and uses memory proportional to the number of distinct attribute combinations rather than the raw dataset size.

Building on this search engine, the paper introduces RADREG, a regression technique that automatically discovers high‑order interactions among real‑valued attributes. RADREG repeatedly invokes RADSEARCH to locate the subset of attributes that yields the greatest reduction in residual error, then adds the corresponding interaction term to the model. Experiments demonstrate that RADREG achieves 5‑8 % lower root‑mean‑square error compared with standard linear regression, Lasso, and tree‑based methods, especially when the underlying data contain complex nonlinear dependencies.

Beyond performance numbers, the work explores the practical implications of exact optimality. The authors argue that, when the cost of a suboptimal model is high (e.g., in scientific discovery or high‑stakes decision making), the modest extra computation required by RADSEARCH is justified. Moreover, the tree‑based aggregation and bound‑driven pruning framework is presented as a general design pattern that could be adapted to other combinatorial search problems involving continuous attributes.

In summary, the paper makes three major contributions: (1) a theoretically sound, globally optimal search algorithm for real‑valued attribute subsets with provably low overhead; (2) a comprehensive empirical comparison that validates its superiority over several state‑of‑the‑art methods; and (3) the RADREG regression model that leverages RADSEARCH to capture high‑order nonlinear interactions, delivering measurable gains in predictive accuracy. The proposed techniques have broad relevance for data mining, machine learning, and statistical optimization tasks where exhaustive yet efficient exploration of the attribute space is required.