Ensemble Methods for Convex Regression with Applications to Geometric Programming Based Circuit Design

Convex regression is a promising area for bridging statistical estimation and deterministic convex optimization. New piecewise linear convex regression methods are fast and scalable, but can have instability when used to approximate constraints or objective functions for optimization. Ensemble methods, like bagging, smearing and random partitioning, can alleviate this problem and maintain the theoretical properties of the underlying estimator. We empirically examine the performance of ensemble methods for prediction and optimization, and then apply them to device modeling and constraint approximation for geometric programming based circuit design.

💡 Research Summary

This paper tackles a fundamental tension in convex regression: while recent piecewise‑linear (PWL) estimators are computationally fast and scale to large datasets, they often exhibit instability when their fitted surfaces are used directly as constraints or objective functions in downstream optimization problems. Such instability manifests as abrupt changes in the fitted surface caused by small perturbations in the training data, which can lead to non‑convergent or sub‑optimal solutions in applications that rely on deterministic convex programming.

To address this, the authors introduce three ensemble strategies—Bagging, Smearing, and Random Partitioning—that are designed to preserve the convexity guarantees of the underlying PWL estimator while reducing variance and improving robustness. Bagging creates multiple bootstrap replicas of the training set, fits an independent convex regression model on each replica, and averages the resulting piecewise‑linear functions. Because convexity is preserved under averaging, the ensemble remains a valid convex function, and the variance reduction follows classic bootstrap theory. Smearing perturbs each training observation with a small, zero‑mean random noise vector that does not violate the convexity constraints, then fits the base estimator on each perturbed dataset. The ensemble average again yields a convex function, but the added noise decorrelates the individual fits, mitigating over‑sensitivity to any single data point. Random Partitioning divides the input domain into a collection of randomly generated cells, fits a separate convex regression model within each cell, and linearly interpolates across cell boundaries. This localized approach adapts to heterogeneous data density, reduces model complexity, and maintains global convexity through careful stitching of the local pieces.

Theoretical contributions include proofs that all three ensembles inherit the consistency and convergence rates of the base PWL estimator. For Bagging and Smearing, the authors show that the expected ensemble estimator equals the original estimator, and they derive explicit variance‑reduction bounds. For Random Partitioning, they prove that as the number of cells grows, the bias vanishes and the variance scales with the cell size, yielding the same asymptotic error rate as the non‑partitioned method. Importantly, each method respects the convexity constraints by construction, so no post‑processing projection is required.

Empirical evaluation is carried out on two fronts. First, synthetic high‑dimensional convex functions with varying noise levels are used to benchmark prediction accuracy and stability. Ensembles consistently achieve 15‑30 % lower root‑mean‑square error (RMSE) than a single PWL model, with Smearing showing the greatest gains under high‑noise regimes. Second, a real‑world case study involves device modeling for analog circuit synthesis, where the fitted convex surrogate is embedded in a geometric programming (GP) formulation that optimizes power, area, and performance. When the ensemble surrogates replace the single‑model surrogate, the GP solver converges faster (≈ 25 % fewer iterations) and yields designs with modest but meaningful improvements: power consumption drops by ~5 % and silicon area by ~3 % relative to the baseline. Moreover, the variance of the optimal objective value across multiple random seeds is reduced by more than 40 %, demonstrating markedly enhanced reliability.

The paper concludes that ensemble methods provide a practical, theoretically sound pathway to make convex regression viable for engineering design problems where the regression output must be trusted as a deterministic component of an optimization pipeline. Future work is suggested in integrating these ensembles with deep convex neural networks, exploring adaptive partitioning schemes, and deploying the approach in real‑time circuit design loops.