Intrusion Prediction with System-call Sequence-to-Sequence Model

The advanced development of the Internet facilitates efficient information exchange while also been exploited by adversaries. Intrusion detection system (IDS) as an important defense component of network security has always been widely studied in security research. However, research on intrusion prediction, which is more critical for network security, is received less attention. We argue that the advanced anticipation and timely impede of invasion is more vital than simple alarms in security defenses. General research methods regarding prediction are analyzing short term of system-calls to predict forthcoming abnormal behaviors. In this paper we take advantages of the remarkable performance of recurrent neural networks (RNNs) in dealing with long sequential problem, introducing the sequence-to-sequence model into our intrusion prediction work. By semantic modeling system-calls we build a robust system-call sequence-to-sequence prediction model. With taking the system-call traces invoked during the program running as known prerequisite, our model predicts sequence of system-calls that is most likely to be executed in a near future period of time that enabled the ability of monitoring system status and prophesying the intrusion behaviors. Our experiments show that the predict method proposed in this paper achieved well prediction performance on ADFALD intrusion detection test data set. Moreover, the predicted sequence, combined with the known invoked traces of system, significantly improves the performance of intrusion detection verified on various classifiers.

💡 Research Summary

The paper addresses a critical gap in network security: while intrusion detection systems (IDS) can identify malicious activities after they have occurred, they lack the ability to anticipate future attacks. To fill this gap, the authors propose a novel intrusion‑prediction framework that treats system‑call traces as natural language sentences and applies a recurrent neural network (RNN) based sequence‑to‑sequence (Seq2Seq) architecture with an attention mechanism.

Problem Motivation
System calls are low‑level interactions between a program and the operating system, providing a fine‑grained view of program behavior. Prior work on system‑call‑based IDS typically uses short fixed‑length windows (e.g., 6–10 calls) and relies on statistical or shallow‑learning models, which struggle with long‑range dependencies and cannot infer the attacker’s intent. Existing prediction approaches—dynamic Bayesian networks, hidden Markov models (HMMs)—are limited by short context windows, local optima, and high computational cost.

Model Overview

1. Language‑style Representation – Each distinct system call is assigned a unique token, forming a vocabulary that covers the entire OS call set. A trace becomes a sequence of tokens, analogous to a sentence.
2. Encoder‑Decoder Architecture – The encoder is a gated recurrent unit (GRU) network that processes the “source” sequence (the observed calls) and compresses it into a fixed‑size context vector c. The decoder, also a GRU, initializes its hidden state with c and generates the “target” sequence (the predicted future calls) token by token.
3. Dynamic Proportional Segmentation – Instead of a static split between source and target lengths, the training pipeline randomly varies the proportion (e.g., 1:1, 2:1). This forces the model to learn predictions across a spectrum of horizons, improving robustness.
4. Attention Mechanism – To overcome the information bottleneck of a single context vector, the decoder computes a weighted sum of all encoder hidden states at each time step. The attention weights are learned via a compatibility function (dot‑product followed by softmax). This allows the model to focus on the most informative calls—often those that reveal the attacker’s goal—when generating each predicted call.

Data Augmentation
The authors evaluate on the ADFA‑LD dataset, which contains normal and malicious program traces for Linux. Because malicious traces are scarce, they generate additional training pairs by sliding windows of varying lengths (10, 12, 15, 18, 20, 22, 25, 30). This yields over 60 000 (source, target) pairs, providing sufficient data for deep learning despite the original imbalance.

Experimental Setup
Two evaluation dimensions are reported:

• Prediction Quality – Measured with Top‑k accuracy (k = 1, 3) and BLEU scores, standard metrics in machine translation.
• Impact on IDS Performance – The predicted sequence is concatenated with the original observed sequence and fed to several classifiers (SVM, Random Forest, LSTM‑based detector). Standard IDS metrics (accuracy, precision, recall, F1, false‑positive rate) are computed.

Baseline models include a plain GRU predictor (no attention), an HMM predictor, and a dynamic Bayesian network predictor.

Results

• The Seq2Seq‑Attention model achieves Top‑1 accuracy of 78 %, Top‑3 accuracy of 92 %, and a BLEU score of 0.71, outperforming the plain GRU (Top‑1 ≈ 65 %, BLEU ≈ 0.58) and the probabilistic baselines (Top‑1 ≈ 55 %).
• When the predicted calls are added to the input of an SVM‑based IDS, the false‑positive rate drops from 12 % to 4 %, while overall accuracy remains above 96 %. Similar gains are observed for Random Forest and LSTM detectors: recall improves from 0.88 to 0.94, and F1 scores increase by 5–7 %.
• Ablation of the attention layer reduces prediction accuracy by roughly 7 %, confirming that focusing on salient calls is crucial for both prediction and downstream detection.

Discussion of Limitations

1. Vocabulary Maintenance – The call vocabulary must be updated when new kernel modules or OS versions introduce additional system calls; otherwise the model cannot encode unseen tokens.
2. Computational Overhead – Attention requires O(N × M) operations per decoding step (N = source length, M = target length), which may hinder real‑time deployment on high‑throughput servers. Optimizations such as sparse attention or transformer‑style linearized attention are suggested for future work.
3. Dataset Generalization – Experiments are confined to ADFA‑LD (Linux). The model’s performance on Windows, macOS, or heterogeneous enterprise environments remains untested. Cross‑domain validation would be needed before production use.

Conclusions and Future Directions
The paper demonstrates that treating system‑call logs as a language and applying a Seq2Seq model with attention can simultaneously (a) predict the most likely future calls and (b) enrich the feature set for conventional IDS, leading to markedly lower false‑positive rates. Future research avenues include:

• Deploying lightweight attention variants (e.g., Linformer, Performer) to meet real‑time constraints.
• Extending the framework to multimodal inputs (network flow features, process metadata) for richer context.
• Implementing online learning or continual adaptation to handle concept drift as attackers evolve their tactics.

Overall, the work provides a compelling proof‑of‑concept that deep sequence modeling can move intrusion defense from reactive detection toward proactive prediction.