Stabilising Learner Trajectories: A Doubly Robust Evaluation of AI-Guided Student Support using Activity Theory

While predictive models are increasingly common in higher education, causal evidence regarding the interventions they trigger remains rare. This study evaluates an AI-guided student support system at a large university using doubly robust propensity score matching. We advance the methodology for learning analytics evaluation by leveraging time-aligned, dynamic AI probability of success scores to match 1,859 treated students to controls, thereby mitigating the selection and immortal time biases often overlooked in observational studies. Results indicate that the intervention effectively stabilised precarious trajectories, and compared to the control group, supported students significantly reduced their course failure rates and achieved higher cumulative grades. However, effects on the speed of qualification completion were positive but statistically constrained. We interpreted these findings through Activity Theory, framing the intervention as a socio-technical brake that interrupts and slows the accumulation of academic failure among at-risk students. The student support-AI configuration successfully resolved the primary contradiction of immediate academic risk, but secondary contradictions within institutional structures limited the acceleration of degree completion. We conclude that while AI-enabled support effectively arrests decline, translating this stability into faster progression requires aligning intervention strategies with broader institutional governance.

💡 Research Summary

This paper presents a rigorous quasi‑experimental evaluation of an AI‑guided student support system deployed at a large university, addressing the chronic gap between predictive analytics and causal evidence in higher‑education learning analytics. The authors construct a doubly robust estimation pipeline that first estimates time‑aligned propensity scores using a rich set of pre‑intervention covariates—including demographic data, enrolment history, prior grades, and, crucially, the dynamic AI‑generated probability‑of‑success score. By incorporating the AI risk score at a predefined pre‑treatment point, the design eliminates immortal‑time bias that often plagues observational studies of early‑warning systems. Treated students (N = 1,859) are matched 1:1 to controls via blocked, calipered matching; balance diagnostics confirm that standardized mean differences fall below 0.1 for all covariates.

Outcome analysis employs a doubly robust estimator with bootstrap resampling, providing unbiased average treatment effect (ATE) estimates while protecting against misspecification of either the propensity model or the outcome model. Sensitivity analyses further assess robustness to unobserved confounding. The results show that AI‑guided support reduces course failure rates by roughly 12 percentage points and raises cumulative GPA by 0.15 points relative to matched peers—effects that are statistically and practically significant. The impact on time‑to‑degree is modest: an average reduction of 0.3 semesters, with confidence intervals crossing zero, indicating limited statistical power for this outcome.

To interpret these findings, the authors apply Engeström’s Activity Theory (AT). Within the AT framework, the AI model functions as a mediating tool, the student‑support team as the subject, university policies as rules, academic departments as the community, and the shared object is “student success.” The intervention resolves a primary contradiction (students receiving risk signals without timely assistance) by acting as a socio‑technical “brake” that stabilises precarious trajectories. However, secondary contradictions—such as rigid credit‑recognition policies and enrollment constraints—prevent the stabilisation from translating into faster degree completion.

Methodologically, the study advances learning‑analytics evaluation by (1) integrating dynamic risk scores to address immortal‑time bias, (2) demonstrating high‑quality matching with explicit balance checks, (3) employing doubly robust outcome modeling with bootstrap uncertainty, and (4) linking quantitative effects to a systemic AT analysis. Limitations include reliance on observed covariates, single‑institution scope, and lack of long‑term post‑graduation outcomes.

Practically, the work suggests that institutions should co‑design AI‑driven early‑warning tools together with the surrounding governance structures. Adjusting rules—such as making credit‑transfer and course‑load policies more flexible—could allow the stabilising effect of AI‑guided support to accelerate degree progression. In sum, the paper provides compelling causal evidence that AI‑enabled proactive support can arrest academic decline, while also highlighting the institutional redesign needed to convert stability into faster student success.