Classification by Set Cover: The Prototype Vector Machine

We introduce a new nearest-prototype classifier, the prototype vector machine (PVM). It arises from a combinatorial optimization problem which we cast as a variant of the set cover problem. We propose two algorithms for approximating its solution. The PVM selects a relatively small number of representative points which can then be used for classification. It contains 1-NN as a special case. The method is compatible with any dissimilarity measure, making it amenable to situations in which the data are not embedded in an underlying feature space or in which using a non-Euclidean metric is desirable. Indeed, we demonstrate on the much studied ZIP code data how the PVM can reap the benefits of a problem-specific metric. In this example, the PVM outperforms the highly successful 1-NN with tangent distance, and does so retaining fewer than half of the data points. This example highlights the strengths of the PVM in yielding a low-error, highly interpretable model. Additionally, we apply the PVM to a protein classification problem in which a kernel-based distance is used.

💡 Research Summary

The paper introduces the Prototype Vector Machine (PVM), a novel nearest‑prototype classifier that is derived from a combinatorial optimization formulation equivalent to a variant of the classic set‑cover problem. The authors start by formalizing the learning task: given a training set X = {x₁,…,xₙ} and a dissimilarity measure d(·,·) (which may be non‑Euclidean or kernel‑induced), the goal is to select a small subset P ⊆ X of “prototype” points such that every training example lies within a predefined radius r of at least one prototype belonging to the same class. This requirement is encoded as a binary integer program with variables zᵢ indicating whether xᵢ is chosen as a prototype and variables yᵢⱼ indicating whether prototype i covers example j. The objective balances the cost of selecting prototypes against the cost of covering examples, and the constraints enforce (i) each example must be covered by at least one prototype of its own class and (ii) a prototype can cover only if it has been selected. Because the problem is NP‑hard (it reduces to set cover), the paper does not seek an exact solution but proposes two practical approximation schemes.

The first scheme is a greedy set‑cover algorithm. At each iteration the algorithm selects the candidate prototype that covers the largest number of currently uncovered training points (respecting class labels). After selection, all points covered by that prototype are removed from the uncovered set, and the process repeats until every point is covered. This method inherits the classic logarithmic approximation guarantee of greedy set‑cover and is computationally simple: each iteration requires scanning all candidate prototypes and counting uncovered points, leading to O(n²) time in the worst case.

The second scheme is based on Lagrangian relaxation. The integer constraints are relaxed by introducing Lagrange multipliers for the covering constraints, yielding a continuous dual problem. The dual is optimized using sub‑gradient descent, producing fractional values for the prototype variables. A deterministic rounding step then converts these fractional values into a binary prototype set. This approach can exploit problem‑specific cost structures (e.g., weighted prototypes) and often yields a smaller prototype set than the greedy method, at the expense of a more involved optimization routine.

A key contribution of PVM is its metric‑agnostic nature. Because the formulation only requires a dissimilarity function, any distance—Euclidean, tangent distance, dynamic time warping, or kernel‑induced distances—can be plugged in without modification. This flexibility is crucial for domains where data are not naturally embedded in a vector space (e.g., strings, graphs, biological sequences) or where a domain‑specific metric dramatically improves discrimination.

The authors evaluate PVM on two benchmark problems. The first is the well‑studied ZIP code handwritten digit dataset (16×16 pixel images of digits 0–9). Using tangent distance, 1‑Nearest Neighbor (1‑NN) typically achieves about 2.5 % error. PVM, with the greedy algorithm, selects roughly 45 % of the original 2000 training images as prototypes and reduces the error to under 1.8 %. Thus, PVM attains a better trade‑off between model size and accuracy, and the selected prototypes are visually interpretable representatives of each digit class.

The second experiment concerns protein sequence classification. Here the authors employ a spectrum kernel to define a similarity measure between amino‑acid strings. Conventional approaches (full‑kernel SVM or 1‑NN) require storing the entire kernel matrix, which is memory‑intensive for large sequence collections. PVM, using the Lagrangian‑relaxation algorithm, retains less than 30 % of the sequences as prototypes while achieving classification performance comparable to or slightly better than the full‑kernel SVM. This demonstrates that PVM can dramatically reduce storage and prediction time without sacrificing accuracy.

Theoretical analysis shows that 1‑NN is a special case of PVM when the covering radius r is set to zero (each point must cover only itself) and the cost of selecting a prototype is uniform. Consequently, PVM can be viewed as a principled generalization that subsumes 1‑NN while offering model compression and interpretability.

The paper also discusses limitations and future directions. Because the underlying optimization remains NP‑hard, the quality of the solution depends on the chosen approximation algorithm and on dataset characteristics (e.g., class imbalance, distribution of distances). The greedy method may select redundant prototypes in highly clustered regions, whereas the Lagrangian approach can be sensitive to step‑size selection in the sub‑gradient routine. Moreover, the computational cost of evaluating the dissimilarity function dominates runtime when the distance is expensive (e.g., complex kernels). Future work suggested includes developing more scalable approximation schemes (e.g., stochastic or parallel greedy variants), extending PVM to online or streaming settings where prototypes must be updated incrementally, and exploring multi‑label or hierarchical classification scenarios.

In summary, the Prototype Vector Machine offers a compelling blend of combinatorial optimization and distance‑based learning. By framing prototype selection as a set‑cover problem, it provides a systematic way to compress training data, accommodate arbitrary dissimilarities, and retain high classification performance. The empirical results on image and biological sequence data illustrate that PVM can outperform strong baselines such as 1‑NN with sophisticated metrics while using substantially fewer training points, thereby delivering models that are both accurate and interpretable.