A Multi-objective Exploratory Procedure for Regression Model Selection

Variable selection is recognized as one of the most critical steps in statistical modeling. The problems encountered in engineering and social sciences are commonly characterized by over-abundance of explanatory variables, non-linearities and unknown interdependencies between the regressors. An added difficulty is that the analysts may have little or no prior knowledge on the relative importance of the variables. To provide a robust method for model selection, this paper introduces the Multi-objective Genetic Algorithm for Variable Selection (MOGA-VS) that provides the user with an optimal set of regression models for a given data-set. The algorithm considers the regression problem as a two objective task, and explores the Pareto-optimal (best subset) models by preferring those models over the other which have less number of regression coefficients and better goodness of fit. The model exploration can be performed based on in-sample or generalization error minimization. The model selection is proposed to be performed in two steps. First, we generate the frontier of Pareto-optimal regression models by eliminating the dominated models without any user intervention. Second, a decision making process is executed which allows the user to choose the most preferred model using visualisations and simple metrics. The method has been evaluated on a recently published real dataset on Communities and Crime within United States.

💡 Research Summary

The paper addresses the long‑standing problem of variable selection in regression, where analysts often face an abundance of candidate predictors, possible non‑linear relationships, and unknown inter‑dependencies. Traditional approaches—such as information‑criterion based stepwise procedures (AIC, BIC), exhaustive best‑subset searches, or heuristic genetic algorithms—typically collapse the problem to a single objective (e.g., minimizing a penalized loss) and consequently return a single “best” model. This single‑model focus can be misleading because the trade‑off between model complexity (parsimony) and goodness‑of‑fit is inherently multi‑objective, and different users may have different preferences regarding that trade‑off.

To overcome these limitations, the authors formulate regression model selection as a bi‑objective optimization problem. The first objective, ϕ₁, measures model complexity by counting the number of selected predictors; the second objective, ϕ₂, measures empirical risk using the mean‑squared error (MSE) on the training data. A model f dominates another model g if it is no worse on both objectives and strictly better on at least one. The set of non‑dominated solutions constitutes the Pareto front, which represents the “best‑subset” of models across all possible complexity‑fit compromises.

The proposed algorithm, Multi‑Objective Genetic Algorithm for Variable Selection (MOGA‑VS), employs an evolutionary search inspired by NSGA‑II. An initial population of binary chromosomes encodes candidate subsets of variables. Each chromosome is evaluated for ϕ₁ and ϕ₂ by fitting a linear regression model and computing its MSE. The population is then sorted into Pareto fronts; within each front, a crowding‑distance metric preserves diversity. Standard genetic operators—one‑point crossover and bit‑flip mutation—generate offspring, and elitist replacement ensures that the best non‑dominated solutions survive across generations. The process repeats until a stopping criterion (maximum generations or convergence of the front) is met.

A distinctive feature of MOGA‑VS is the explicit separation of two phases: (1) an automated exploration phase that returns the entire Pareto front without any user intervention, and (2) a decision‑making phase where the analyst examines the front using visual tools (complexity vs. MSE plots) and simple quantitative cues (e.g., elbow points, minimum‑complexity model, minimum‑generalization‑error model). This design empowers users to inject domain knowledge or practical constraints after the algorithm has already identified the full spectrum of optimal trade‑offs.

The authors evaluate MOGA‑VS on three data sets: (i) synthetic data designed to mimic high‑dimensional, noisy settings, (ii) the Boston Housing data set, and (iii) a recently released “Communities and Crime” data set from the United States, which contains dozens of socio‑economic predictors. For each data set, they compare MOGA‑VS against several baselines: AIC/BIC‑guided stepwise selection, forward/backward selection, exhaustive best‑subset (where feasible), and Bayesian Model Averaging (BMA). Performance is assessed using training MSE, validation (or cross‑validated) MSE, model size, and the spread of the Pareto front.

Results consistently show that MOGA‑VS discovers a richer set of non‑dominated models than the baselines. In particular, the Pareto front produced by MOGA‑VS includes models with substantially lower validation MSE while maintaining comparable or smaller model sizes. When the algorithm is modified to directly minimize an estimate of generalization error (instead of training MSE), it further reduces over‑fitting risk, outperforming the single‑objective GA approaches that optimize only an information criterion. The authors also note that the algorithm scales reasonably well for moderate numbers of predictors (up to a few dozen), but computational cost grows with dimensionality, as is typical for evolutionary methods.

The paper discusses several limitations. First, when the Pareto front contains a very large number of points, visual inspection and final model selection can become cumbersome, suggesting a need for automated front‑compression or clustering techniques. Second, the current implementation assumes linear regression and MSE as the loss; extending the framework to generalized linear models, non‑linear kernels, or alternative loss functions would broaden applicability. Third, the evolutionary search, while effective, is stochastic and may require careful tuning of population size, crossover/mutation rates, and termination criteria for very high‑dimensional problems.

In conclusion, the authors present MOGA‑VS as a robust, flexible framework that reframes regression variable selection as a genuine multi‑objective problem, delivering a full Pareto frontier of candidate models and allowing analysts to make informed, preference‑driven choices. Future work is outlined to include (a) incorporation of additional objectives such as sparsity‑inducing penalties or interpretability scores, (b) development of automated decision‑support tools for navigating large Pareto sets, and (c) parallel or distributed implementations to handle truly large‑scale data. The study demonstrates that embracing multi‑objective optimization can substantially enrich the model‑selection process beyond what traditional single‑criterion methods offer.