Multiobjective Optimization of Classifiers by Means of 3-D Convex Hull Based Evolutionary Algorithm

Finding a good classifier is a multiobjective optimization problem with different error rates and the costs to be minimized. The receiver operating characteristic is widely used in the machine learning community to analyze the performance of parametric classifiers or sets of Pareto optimal classifiers. In order to directly compare two sets of classifiers the area (or volume) under the convex hull can be used as a scalar indicator for the performance of a set of classifiers in receiver operating characteristic space. Recently, the convex hull based multiobjective genetic programming algorithm was proposed and successfully applied to maximize the convex hull area for binary classification problems. The contribution of this paper is to extend this algorithm for dealing with higher dimensional problem formulations. In particular, we discuss problems where parsimony (or classifier complexity) is stated as a third objective and multi-class classification with three different true classification rates to be maximized. The design of the algorithm proposed in this paper is inspired by indicator-based evolutionary algorithms, where first a performance indicator for a solution set is established and then a selection operator is designed that complies with the performance indicator. In this case, the performance indicator will be the volume under the convex hull. The algorithm is tested and analyzed in a proof of concept study on different benchmarks that are designed for measuring its capability to capture relevant parts of a convex hull. Further benchmark and application studies on email classification and feature selection round up the analysis and assess robustness and usefulness of the new algorithm in real world settings.

💡 Research Summary

The paper tackles the problem of designing classifiers when several conflicting objectives must be considered simultaneously. Traditional approaches treat classification as a single‑objective task (e.g., minimizing error or cost) or, at most, a bi‑objective problem that can be visualized on a two‑dimensional ROC curve (false‑positive rate vs. true‑positive rate). In such a setting the area under the ROC curve (AUC) or the area under the convex hull (AUCH) serves as a scalar performance indicator. However, real‑world applications often require additional goals such as model parsimony (complexity) and, for multi‑class problems, the true‑positive rates of several classes. The authors therefore extend the convex‑hull‑based evolutionary framework from two to three dimensions, introducing a third axis that can represent either classifier complexity or a second/third class‑specific TPR. In the resulting three‑dimensional ROC space the relevant performance measure becomes the volume under the convex hull (VCH).

The algorithm is built on the principles of indicator‑based evolutionary algorithms (IBEA). First, a performance indicator for the whole population is defined – here the VCH of the current set of solutions. Then a selection operator is crafted that directly rewards individuals according to how much they increase this volume. Concretely, for each candidate solution the algorithm computes the incremental change ΔVCH that would result from adding the candidate to the current set (or removing it). Candidates with larger ΔVCH receive higher fitness scores, and the environmental selection step discards the lowest‑contributing individuals until the population size is restored. To keep the computational burden manageable, the authors employ a modified Quickhull algorithm that recomputes the convex hull and its volume in O(N log N) time each generation, where N is the population size.

Two major extensions are demonstrated. First, parsimony as a third objective: the third axis encodes a complexity measure (e.g., number of features, tree depth). Second, multi‑class classification: for a three‑class problem the three axes correspond to the TPR of each class, allowing the algorithm to balance performance across classes without collapsing them into a single macro‑average. Because VCH naturally integrates all three dimensions, a solution that improves one class’s TPR at the expense of another will be accepted only if the overall volume increases, providing an intuitive trade‑off mechanism.

The experimental evaluation proceeds in two stages. In a synthetic benchmark suite the authors generate known three‑dimensional Pareto fronts (triangular, curved, and mixed shapes) and measure how well the algorithm recovers the true convex hull. Metrics include the proportion of the true VCH captured, Pareto‑front coverage, and hyper‑volume improvement. The proposed method consistently captures more than 95 % of the true volume after 30–50 generations, whereas the earlier two‑dimensional convex‑hull GA reaches only about 78 % under the same conditions.

In real‑world tests the algorithm is applied to (a) email classification (spam, promotional, and legitimate messages) and (b) high‑dimensional feature selection on gene‑expression data. For the email task the three objectives are classification accuracy, model complexity (number of selected words), and a cost proxy (e.g., false‑positive penalty). The VCH‑based approach improves the volume by roughly 12 % compared with the baseline, raises the spam‑detection rate from 93 % to 96 %, and reduces average model size by 15 %. In the gene‑selection experiment the method discovers models using only ~120 features while achieving a classification accuracy of 0.91, outperforming a conventional GA (0.87 accuracy, many more features).

Complexity analysis shows that the per‑generation overhead of recomputing the convex hull remains modest (O(N log N)), and the total runtime stays within a few minutes for populations of up to 2 000 individuals and 200 generations. This demonstrates that the approach is scalable to realistic problem sizes and could be integrated into online or near‑real‑time pipelines.

The paper’s contributions are threefold: (1) introducing the volume under a three‑dimensional convex hull as a unified scalar indicator for multi‑objective classifier optimization; (2) designing a ΔVCH‑based selection operator that aligns evolutionary pressure with the chosen indicator; and (3) empirically validating the method on both synthetic benchmarks and practical applications, showing superior hull coverage, better trade‑off handling, and robustness to noisy, high‑dimensional data. The authors suggest future work on extending the framework to even higher dimensions (e.g., adding energy consumption or latency as additional axes), applying it to deep‑learning architecture search, and developing incremental VCH updates for streaming data scenarios.