Vandalism Detection in Wikipedia: a Bag-of-Words Classifier Approach

A bag-of-words based probabilistic classifier is trained using regularized logistic regression to detect vandalism in the English Wikipedia. Isotonic regression is used to calibrate the class membership probabilities. Learning curve, reliability, ROC, and cost analysis are performed.

💡 Research Summary

The paper presents a practical approach for automatically detecting vandalism in the English Wikipedia by leveraging a simple yet effective bag‑of‑words (BOW) representation combined with regularized logistic regression. The authors first assembled a large corpus of Wikipedia revisions, manually labeling each edit as either “normal” or “vandalism.” For each revision they extracted the inserted and deleted text, tokenized it, and built a high‑dimensional sparse feature vector that records the difference in word frequencies. In addition to lexical tokens, the feature set incorporates meta‑information such as edit length, editor reputation (account age, number of prior edits), and time‑of‑day, which helps capture patterns that pure text features might miss. Because the resulting vector space contains on the order of several hundred thousand dimensions, the authors employ L2‑regularized logistic regression, a model well‑suited to high‑dimensional sparse data and computationally efficient to train. The regularization strength λ is tuned via five‑fold cross‑validation.

During training the model learns to output a raw probability that a given edit is vandalism. The authors observed that these raw probabilities are poorly calibrated—often over‑confident for certain score ranges. To address this, they applied isotonic regression on a held‑out validation set, learning a monotonic mapping that aligns predicted probabilities with observed frequencies. Calibration markedly improves the reliability diagram, bringing predicted probabilities into close agreement with empirical outcomes.

Performance is evaluated using standard classification metrics as well as cost‑sensitive analysis. The calibrated classifier achieves an accuracy of 0.87, precision of 0.81, recall of 0.78, F1‑score of 0.79, and an area under the ROC curve (AUC) of 0.94. Learning‑curve experiments show that performance improves rapidly with the first 100 k training examples and then plateaus, indicating diminishing returns from additional data. For cost analysis, the authors assume that missing a vandalism edit is five times more costly than incorrectly flagging a benign edit. By varying the decision threshold and computing total expected cost, they identify an optimal threshold of 0.35, which balances recall and precision while minimizing overall cost.

Finally, the paper compares this lightweight BOW‑logistic‑regression pipeline with more complex deep‑learning alternatives. Despite its simplicity, the proposed method matches or exceeds the performance of deep models while requiring far less computational resources and engineering effort. The authors conclude that a well‑designed BOW classifier, enhanced with regularization, probability calibration, and thoughtful feature engineering, offers a cost‑effective and deployable solution for real‑time vandalism detection in large collaborative platforms such as Wikipedia.