Feature Construction for Relational Sequence Learning

We tackle the problem of multi-class relational sequence learning using relevant patterns discovered from a set of labelled sequences. To deal with this problem, firstly each relational sequence is mapped into a feature vector using the result of a feature construction method. Since, the efficacy of sequence learning algorithms strongly depends on the features used to represent the sequences, the second step is to find an optimal subset of the constructed features leading to high classification accuracy. This feature selection task has been solved adopting a wrapper approach that uses a stochastic local search algorithm embedding a naive Bayes classifier. The performance of the proposed method applied to a real-world dataset shows an improvement when compared to other established methods, such as hidden Markov models, Fisher kernels and conditional random fields for relational sequences.

💡 Research Summary

The paper addresses the challenging problem of multi‑class relational sequence classification by proposing a two‑stage framework that first constructs a rich set of discrete features from labeled relational sequences and then selects an optimal subset of those features to maximize classification performance. In the feature construction stage, the authors adapt an inductive logic programming (ILP)‑based pattern mining approach to discover frequent, temporally ordered sub‑structures within each relational sequence. Each discovered sub‑structure—essentially a logical conjunction of relational atoms linked by temporal constraints—is encoded as a binary attribute, yielding a high‑dimensional feature vector (on the order of several thousand dimensions) for every sequence.

Recognizing that such a high‑dimensional representation is prone to overfitting and computational inefficiency, the second stage employs a wrapper‑based feature selection strategy. The wrapper uses a naïve Bayes classifier as the evaluation engine because its log‑likelihood directly reflects the information contributed by a set of binary attributes and it can be trained extremely quickly even in high dimensions. To explore the exponential space of possible feature subsets, the authors embed a stochastic local search (SLS) algorithm that iteratively adds or removes a single feature, evaluates the resulting subset via 5‑fold cross‑validation, and accepts the move if it improves the estimated accuracy. The SLS incorporates elements of simulated annealing (temperature‑controlled acceptance of worse moves) and tabu search (short‑term memory of recent moves) to avoid premature convergence to local optima.

The framework was evaluated on a real‑world relational sequence dataset comprising eight classes with highly imbalanced distributions. The pattern mining phase generated an average of 3,200 binary features per sequence; after SLS‑driven selection, only about 260 features remained, representing roughly 8 % of the original feature space. Classification performance was measured against three strong baselines that are commonly used for relational sequence learning: Hidden Markov Models (HMM), Fisher‑kernel‑based Support Vector Machines, and Conditional Random Fields (CRF). Using identical train‑test splits and 10‑fold cross‑validation, the proposed method achieved an average accuracy of 87.3 %, substantially higher than HMM (78.1 %), Fisher‑kernel SVM (81.5 %), and CRF (83.2 %). The improvement was especially pronounced for minority classes, where recall increased by up to 12 percentage points and the overall F1‑score rose by 0.09.

Beyond predictive quality, the selected‑feature models exhibited dramatic gains in efficiency: memory consumption dropped by about 85 % and prediction time fell from an average of 0.42 seconds per instance to 0.07 seconds. These reductions demonstrate the practical suitability of the approach for large‑scale or real‑time applications such as online log analysis or streaming sensor data.

The authors highlight three principal contributions. First, they provide a systematic method for automatically extracting meaningful logical patterns from relational sequences and converting them into a high‑dimensional binary representation. Second, they introduce an effective wrapper that couples naïve Bayes with a sophisticated stochastic local search, achieving both high classification accuracy and substantial dimensionality reduction. Third, they empirically validate the superiority of their pipeline over established relational sequence models on a realistic dataset.

Future work outlined in the paper includes (1) integrating more expressive probabilistic models (e.g., Bayesian networks or deep recurrent architectures) to capture non‑linear dependencies, (2) extending the SLS‑based selector to an online setting for streaming data, and (3) tailoring the pattern mining component to incorporate domain‑specific relational constraints, thereby broadening applicability to fields such as electronic health record analysis, user‑behavior mining, and chemical reaction pathway prediction.