A neural network approach to ordinal regression

Ordinal regression is an important type of learning, which has properties of both classification and regression. Here we describe a simple and effective approach to adapt a traditional neural network to learn ordinal categories. Our approach is a generalization of the perceptron method for ordinal regression. On several benchmark datasets, our method (NNRank) outperforms a neural network classification method. Compared with the ordinal regression methods using Gaussian processes and support vector machines, NNRank achieves comparable performance. Moreover, NNRank has the advantages of traditional neural networks: learning in both online and batch modes, handling very large training datasets, and making rapid predictions. These features make NNRank a useful and complementary tool for large-scale data processing tasks such as information retrieval, web page ranking, collaborative filtering, and protein ranking in Bioinformatics.

💡 Research Summary

The paper introduces NNRank, a neural‑network‑based method for ordinal regression that adapts a conventional multilayer perceptron to predict ordered categories. Unlike standard multiclass classification, where a softmax layer yields a probability distribution over mutually exclusive classes, ordinal regression requires the model to respect the inherent ordering among labels. NNRank addresses this by replacing the softmax output with K‑1 binary neurons (for K ordered classes). Each neuron learns to answer the question “Is the target value greater than threshold t?” where t ranges from the first to the (K‑1)‑th boundary. The final predicted class is obtained by counting how many of these binary outputs are “1”, i.e., the number of thresholds exceeded.

Training uses the usual back‑propagation algorithm. For each binary output a binary cross‑entropy loss is computed, and the total loss is the sum of these K‑1 terms. Because the loss is decomposed into independent binary components, the gradient with respect to each weight is simply the sum of the gradients from each output neuron, allowing the model to be trained with any standard optimizer (SGD, momentum, Adam, etc.) in either batch or online mode. This design preserves all the practical advantages of traditional neural networks: scalability to very large datasets, fast inference, and straightforward integration with existing deep‑learning frameworks.

The authors evaluate NNRank on several benchmark ordinal datasets from the UCI repository (e.g., wine quality, car evaluation, abalone) and on larger‑scale ranking tasks such as web‑page ranking and collaborative‑filtering style recommendation. They compare four methods: (1) NNRank, (2) a conventional neural network with a softmax classifier, (3) ordinal support‑vector machines (Rank‑SVM), and (4) Gaussian‑process ordinal regression. Performance is measured with classification accuracy, mean absolute error (MAE), and ranking‑specific metrics such as NDCG.

Results show that NNRank consistently outperforms the softmax‑based neural classifier, achieving higher accuracy and lower MAE across all datasets. When compared with Rank‑SVM and Gaussian‑process models, NNRank attains comparable or slightly better predictive quality while being dramatically faster to train and to predict. Gaussian processes suffer from O(N³) kernel matrix operations, making them impractical for datasets with more than a few thousand instances. Rank‑SVM requires constructing O(N²) pairwise comparisons, which also limits scalability. In contrast, NNRank’s computational cost grows linearly with the number of training examples and the number of hidden units, exactly as in ordinary feed‑forward networks.

The paper also discusses limitations. Because NNRank uses one binary output per threshold, the number of output units grows linearly with the number of ordinal levels; for problems with hundreds of ordered categories this can increase memory usage and training time. Moreover, the current loss treats each binary output independently, ignoring potential correlations among thresholds that could carry useful information about the underlying ordering. The authors suggest future work such as introducing structured regularization that couples neighboring thresholds, or designing a unified ordinal loss (e.g., ordinal cross‑entropy or hinge‑type loss) that directly optimizes the ordering constraint.

In conclusion, NNRank offers a simple yet effective neural‑network solution for ordinal regression. It leverages the well‑established back‑propagation machinery, works in both batch and online settings, and scales to large‑scale data typical of information‑retrieval, web ranking, collaborative filtering, and bioinformatics applications (e.g., protein ranking). By explicitly modeling the ordinal structure while retaining the computational efficiency of standard neural networks, NNRank serves as a valuable complement to existing ordinal regression techniques such as Rank‑SVM and Gaussian‑process models.