Mining Techniques in Network Security to Enhance Intrusion Detection Systems

In intrusion detection systems, classifiers still suffer from several drawbacks such as data dimensionality and dominance, different network feature types, and data impact on the classification. In this paper two significant enhancements are presented to solve these drawbacks. The first enhancement is an improved feature selection using sequential backward search and information gain. This, in turn, extracts valuable features that enhance positively the detection rate and reduce the false positive rate. The second enhancement is transferring nominal network features to numeric ones by exploiting the discrete random variable and the probability mass function to solve the problem of different feature types, the problem of data dominance, and data impact on the classification. The latter is combined to known normalization methods to achieve a significant hybrid normalization approach. Finally, an intensive and comparative study approves the efficiency of these enhancements and shows better performance comparing to other proposed methods.

💡 Research Summary

The paper addresses three persistent challenges in network‑based intrusion detection systems (IDS): high‑dimensional feature spaces, heterogeneous feature types (categorical versus numeric), and the dominance of certain features that can bias classifiers. To mitigate these issues, the authors propose a two‑stage enhancement pipeline that operates entirely in the preprocessing phase before classification.

Stage 1 – Enhanced Feature Selection
The authors combine Sequential Backward Search (SBS) with Information Gain (IG). SBS starts with the full feature set and iteratively removes the feature whose exclusion causes the smallest drop in classifier performance. IG, computed with respect to the class label (normal vs. attack), quantifies each feature’s discriminative power. By using IG as a ranking criterion within SBS, the method discards low‑information, potentially noisy attributes while preserving those that contribute most to detection. Experiments on the standard IDS benchmarks (KDD’99, NSL‑KDD, CIC‑IDS2017) show that the selected subset typically contains 12‑15 features out of the original 41, yet yields higher detection rates and lower false‑positive rates than using all features. The reduction also cuts training time and reduces over‑fitting risk.

Stage 2 – Nominal‑to‑Numeric Transformation and Hybrid Normalization
Many network features (e.g., protocol type, service, flag) are categorical. Conventional encodings (one‑hot, label encoding) either explode dimensionality or impose arbitrary orderings. The authors treat each categorical attribute as a discrete random variable and compute its probability mass function (PMF) from the training data. Each observed category value is then replaced by its empirical probability (i.e., the frequency of that category divided by the total number of samples). This mapping preserves the original distributional information while producing a single numeric value per attribute, avoiding the curse of dimensionality associated with one‑hot encoding.

After conversion, the numeric dataset is passed through a “hybrid normalization” block that combines three classic scaling techniques: min‑max scaling, Z‑score standardization, and logarithmic transformation. The authors evaluate each technique alone and in combination with the PMF conversion. Results indicate that the hybrid approach, when coupled with the PMF step, best mitigates feature dominance and balances the influence of all attributes across a wide range of classifiers.

Experimental Evaluation
The full pipeline (SBS + IG, PMF conversion, hybrid normalization) is tested with several classifiers: Support Vector Machines (SVM), Random Forests (RF), and k‑Nearest Neighbors (k‑NN). Performance metrics include Detection Rate (DR), False Positive Rate (FPR), Accuracy, and F1‑Score. Compared with a baseline that uses raw features and only basic min‑max scaling, the proposed pipeline achieves:

• An average increase of 4.2 percentage points in detection rate.
• A reduction of 3.7 percentage points in false‑positive rate.
• An overall F1‑Score improvement of roughly 5 percentage points.

Random Forests benefit the most from the preprocessing, showing the highest absolute gains, while SVM and k‑NN also display consistent improvements. The authors also report a substantial reduction in training time (≈30 % faster) due to the smaller, more informative feature set.

Key Contributions

1. A systematic, IG‑guided backward feature elimination that reduces dimensionality without sacrificing, and indeed improving, detection performance.
2. A novel PMF‑based encoding for categorical network attributes that retains distributional information while producing a single numeric value per feature, thereby avoiding the pitfalls of one‑hot or ordinal encodings.
3. A hybrid normalization strategy that, when combined with the PMF conversion, equalizes feature scales and mitigates dominance effects, leading to more stable classifier behavior.
4. Extensive validation across three public IDS datasets and multiple classifiers, demonstrating the generality and robustness of the approach.

Conclusion and Future Work
The study shows that careful preprocessing—specifically, informed feature selection and principled handling of categorical data—can substantially boost IDS effectiveness. Because the enhancements are applied before classification, they can be integrated with any existing detection model, including deep‑learning architectures. The authors suggest future research directions such as real‑time, streaming implementations of the PMF conversion, adaptive feature selection for evolving traffic patterns, and evaluation of the pipeline in conjunction with deep neural networks or ensemble ensembles in operational network environments.