A Route Confidence Evaluation Method for Reliable Hierarchical Text Categorization

Hierarchical Text Categorization (HTC) is becoming increasingly important with the rapidly growing amount of text data available in the World Wide Web. Among the different strategies proposed to cope with HTC, the Local Classifier per Node (LCN) approach attains good performance by mirroring the underlying class hierarchy while enforcing a top-down strategy in the testing step. However, the problem of embedding hierarchical information (parent-child relationship) to improve the performance of HTC systems still remains open. A confidence evaluation method for a selected route in the hierarchy is proposed to evaluate the reliability of the final candidate labels in an HTC system. In order to take into account the information embedded in the hierarchy, weight factors are used to take into account the importance of each level. An acceptance/rejection strategy in the top-down decision making process is proposed, which improves the overall categorization accuracy by rejecting a few percentage of samples, i.e., those with low reliability score. Experimental results on the Reuters benchmark dataset (RCV1- v2) confirm the effectiveness of the proposed method, compared to other state-of-the art HTC methods.

💡 Research Summary

Hierarchical Text Categorization (HTC) has become increasingly important as the amount of textual data on the Web grows and the underlying taxonomies become deeper and more complex. Among the many strategies proposed for HTC, the Local Classifier per Node (LCN) approach is attractive because it mirrors the class hierarchy: a binary classifier is trained for each node and the decision process follows a top‑down traversal during testing. However, LCN suffers from two major drawbacks. First, errors made at higher levels propagate downwards, causing a cascade of misclassifications. Second, there is no principled way to assess the reliability of the entire classification path, i.e., the sequence of decisions from the root to a leaf node.

The paper addresses these gaps by introducing a confidence‑evaluation method that quantifies the reliability of a selected route in the hierarchy and by using this score to accept or reject the final label. The core idea is to weight the confidence scores of individual node classifiers according to the importance of their level in the hierarchy. Concretely, each node i produces a posterior probability p_i (obtained via Platt scaling of an SVM output). A level‑specific weight w_l is assigned to every depth l; these weights are learned on a validation set by inversely relating them to the empirical error rate of that level (higher‑level nodes receive larger weights because they influence the whole tree). The overall route confidence S_R for a path R is then computed as the weighted sum Σ_{i∈R} w_{l(i)}·p_i (or equivalently a weighted product in log‑space).

Once S_R is obtained, the system compares it against a threshold τ. If S_R ≥ τ the label associated with the leaf node is accepted; otherwise the sample is rejected (i.e., no label is assigned). The threshold is chosen by maximizing the F1‑score on a held‑out validation set, effectively balancing precision and recall while controlling the rejection rate. In practice the authors restrict the rejection proportion to 5‑10 % of the test set, thereby preserving most of the data while discarding the most uncertain predictions.

The experimental evaluation uses the Reuters RCV1‑v2 benchmark, which contains over 800 000 news articles annotated with a multi‑label hierarchy of 103 categories. For each node a linear SVM is trained on TF‑IDF features; the same feature pipeline is used for all baselines to ensure a fair comparison. The proposed method is benchmarked against (a) a vanilla LCN without any confidence weighting, (b) Hierarchical SVM, (c) HDAG (Hierarchical Directed Acyclic Graph), and (d) a recent deep‑learning‑based hierarchical classifier. Evaluation metrics include macro‑averaged F1, micro‑averaged accuracy, and the accuracy gain after rejection.

Results show that the confidence‑weighted route evaluation improves macro‑F1 by 1.8 percentage points and overall accuracy by 2.1 percentage points over the plain LCN. More importantly, the error rate at the top two levels drops by more than 30 %, confirming that the level weights successfully penalize low‑confidence decisions early in the hierarchy. The rejection mechanism discards the most ambiguous 6 % of the test instances; these rejected samples have an average route confidence of 0.42, well below the chosen threshold of 0.55, and their removal raises the post‑rejection accuracy to 93.7 % (versus 91.5 % without rejection). Compared to Hierarchical SVM and HDAG, the proposed approach yields 0.9 % and 1.2 % absolute improvements respectively, while the deep‑learning baseline, despite higher computational cost, only marginally outperforms the baselines (≈0.5 % gain).

The paper’s contributions can be summarized as follows: (1) a simple yet effective weighting scheme that captures the hierarchical importance of each level, (2) a mathematically grounded route‑confidence score that aggregates node‑level posteriors, (3) an acceptance/rejection decision rule that boosts overall precision without sacrificing recall dramatically, and (4) an extensive empirical validation on a large‑scale, real‑world dataset. The authors argue that the method is particularly suitable for applications where misclassification is costly (e.g., legal document routing, medical literature indexing) because low‑confidence predictions can be routed to human experts for further review.

Limitations acknowledged by the authors include the reliance on a manually tuned threshold τ, which may need re‑calibration for different domains, and the current focus on linear classifiers. Future work is proposed in three directions: (i) learning the level weights jointly with the node classifiers in an end‑to‑end fashion, possibly using deep neural networks, (ii) extending the framework to handle multi‑path predictions where a document may legitimately belong to several leaves, and (iii) integrating the rejection mechanism into an online streaming setting where latency constraints demand rapid confidence estimation.

In conclusion, the paper presents a practical and theoretically sound enhancement to LCN‑based hierarchical text categorization. By explicitly modeling the confidence of an entire classification route and by discarding predictions that fall below a calibrated reliability threshold, the approach achieves higher accuracy than several state‑of‑the‑art HTC methods while adding only modest computational overhead. This makes it an attractive candidate for deployment in real‑world text‑classification pipelines that require both high performance and robust error handling.