AI-Driven Early Warning Systems for Student Success: Discovering Static Feature Dominance in Temporal Prediction Models

Early identification of at-risk students is critical for effective intervention in online learning environments. This study extends temporal prediction analysis to Week 20 (50% of course duration), comparing Decision Tree and Long Short- Term Memory (LSTM) models across six temporal snapshots. Our analysis reveals that different performance metrics matter at different intervention stages: high recall is critical for early intervention (Weeks 2-4), while balanced precision-recall is important for mid-course resource allocation (Weeks 8-16), and high precision becomes paramount in later stages (Week 20). We demonstrate that static demographic features dominate predictions (68% importance), enabling assessment-free early prediction. The LSTM model achieves 97% recall at Week 2, making it ideal for early intervention, while Decision Tree provides stable balanced performance (78% accuracy) during mid-course. By Week 20, both models converge to similar recall (68%), but LSTM achieves higher precision (90% vs 86%). Our findings also suggest that model selection should depend on intervention timing, and that early signals (Weeks 2-4) are sufficient for reliable initial prediction using primarily demographic and pre-enrollment information.

💡 Research Summary

The paper investigates early‑warning systems for student success in massive open online courses (MOOCs) by extending temporal prediction analysis to the first half of a course (up to Week 20, roughly 50 % of the total duration). Using the Open University Learning Analytics Dataset (OULAD) comprising 32,593 students across seven courses, the authors construct eleven features: eight static demographic variables (gender, region, age band, highest education, disability, module code, presentation code, IMD band), two academic‑history variables (previous attempts, credits studied), and one engagement variable (cumulative VLE clicks). Notably, assessment‑related features (submission counts and average scores) are deliberately excluded from the baseline models, enabling “assessment‑free” prediction that can be deployed before any graded activity occurs.

Two predictive architectures are compared across six temporal snapshots (Weeks 2, 4, 8, 12, 16, 20): a conventional Decision Tree (trained once on the full dataset and evaluated on each snapshot without retraining) and a deep LSTM network (two LSTM layers of 64 and 32 units, dropout 0.3–0.4, class‑weighting for imbalance). The LSTM receives a padded sequence of 84 timesteps (one per day) where each timestep contains the eleven features; shorter snapshots are zero‑padded to maintain a constant input length, while later snapshots truncate the earliest days.

Key findings:

1. Feature importance – Static demographic attributes account for 68 % of total importance, with region (27.66 %), studied credits (13.50 %), and IMD band (11.39 %) leading. Temporal engagement (clicks) contributes only 1.34 %, and assessment features are not used. This demonstrates that early prediction can rely almost entirely on pre‑enrollment information.

2. Decision Tree performance – Baseline accuracy is 76.85 % (precision = 78.04 %, recall = 78.15 %). Across snapshots, accuracy remains stable (≈ 76–78 %). Precision rises from 74 % (Week 2) to 86 % (Week 20) while recall declines from 83 % to 68 %, indicating a shift toward a more conservative classifier as more data accumulates. The model’s interpretability (feature importance, decision paths) is highlighted as valuable for mid‑course resource planning.

3. LSTM performance – Early weeks show very high recall (97 % at Week 2, 93 % at Week 4) but low precision (≈ 53 %). Accuracy improves dramatically from 53.8 % (Week 2) to 80 % (Week 20). By Week 20, precision reaches 90 % while recall aligns with the Decision Tree at 68 %. The early‑stage high recall makes LSTM ideal for interventions where missing a at‑risk student is costly; the later‑stage high precision suits scenarios where educators prefer fewer false alarms.

4. Model‑selection framework – The authors propose a timing‑aware recommendation: use LSTM for early intervention (Weeks 2‑4) to maximize recall; use Decision Tree for mid‑course (Weeks 8‑16) to obtain balanced performance and transparent explanations; revert to LSTM for late‑course (Week 20) when high precision is paramount.

5. Implications – Because static demographics dominate, institutions can launch early‑warning systems without waiting for assessment data, reducing latency and privacy concerns. The findings also surface structural inequities: region and socioeconomic status (IMD) are strong predictors of dropout, suggesting that targeted support may need to address broader contextual factors.

Limitations acknowledged include the exclusion of assessment features (which could improve mid‑to‑late predictions), reliance on a single Decision Tree architecture (no comparison with ensembles like Random Forest or XGBoost), and the computational inefficiency of feeding largely static data into an LSTM. Future work is suggested to incorporate assessment signals, explore richer ensemble models, and design LSTM variants that focus on truly temporal inputs.

Overall, the study contributes a nuanced, evidence‑based approach to early‑warning system design, demonstrating that static demographic information is sufficient for reliable early prediction and that the optimal predictive model depends on the specific intervention horizon and performance metric priorities.