EDM-ARS: A Domain-Specific Multi-Agent System for Automated Educational Data Mining Research

In this technical report, we present the Educational Data Mining Automated Research System (EDM-ARS), a domain-specific multi-agent pipeline that automates end-to-end educational data mining (EDM) research. We conceptualize EDM-ARS as a general framework for domain-aware automated research pipelines, where educational expertise is embedded into each stage of the research lifecycle. As a first instantiation of this framework, we focus on predictive modeling tasks. Within this scope, EDM-ARS orchestrates five specialized LLM-powered agents (ProblemFormulator, DataEngineer, Analyst, Critic, and Writer) through a state-machine coordinator that supports revision loops, checkpoint-based recovery, and sandboxed code execution. Given a research prompt and a dataset, EDM-ARS produces a complete LaTeX manuscript with real Semantic Scholar citations, validated machine learning analyses, and automated methodological peer review. We also provide a detailed description of the system architecture, the three-tier data registry design that encodes educational domain expertise, the specification of each agent, the inter-agent communication protocol, and mechanisms for error-handling and self-correction. Finally, we discuss current limitations, including single-dataset scope and formulaic paper output, and outline a phased roadmap toward causal inference, transfer learning, psychometric, and multi-dataset generalization. EDM-ARS is released as an open-source project to support the educational research community.

💡 Research Summary

The paper introduces EDM‑ARS, a domain‑specific multi‑agent pipeline designed to fully automate the end‑to‑end workflow of educational data mining (EDM) research. The authors argue that while large language model (LLM)‑driven automated research systems such as AI Scientist, CycleResearcher, DeepScientist, and others have demonstrated the feasibility of automating idea generation, experiment execution, and manuscript writing, they generally lack the deep domain knowledge required for rigorous EDM work. Educational datasets, especially large‑scale surveys like the High School Longitudinal Study of 2009 (HSLS:09), contain idiosyncratic features such as negative sentinel codes for missing data, temporal ordering constraints, survey weights, and protected attributes that demand explicit handling to avoid data leakage, biased estimates, and invalid fairness assessments.

EDM‑ARS addresses this gap by embedding domain expertise directly into the system architecture rather than relying on ad‑hoc prompts. The core of the system is a finite‑state‑machine orchestrator that coordinates five specialized LLM‑powered agents: ProblemFormulator, DataEngineer, Analyst, Critic, and Writer. The orchestrator defines nine states (initialized, formulating, engineering, analyzing, critiquing, revising, writing, completed, aborted) and enforces checkpoint‑based serialization at each transition, enabling robust recovery from crashes or API timeouts.

The ProblemFormulator agent queries Semantic Scholar, extracts recent literature, and selects a research question, outcome variable, and predictor set from a three‑tier data registry. The registry stores raw variable definitions, domain‑specific metadata (e.g., meanings of NCES missing‑data codes), and validation rules (e.g., temporal leakage checks, weight usage constraints). The DataEngineer agent automatically generates and runs Python code to clean the raw HSLS:09 data, apply the NCES missing‑value protocol, encode categorical variables, and split the data into training and test sets. The Analyst agent then builds a battery of predictive models—including logistic regression, random forest, XGBoost, ElasticNet, multilayer perceptron, and a stacking ensemble—computes SHAP feature‑importance visualizations, and conducts subgroup fairness analyses (e.g., race, gender, socioeconomic status). All code execution occurs in a sandboxed environment to prevent side effects.

After the analytical stage, the Critic agent evaluates the outputs against a multi‑dimensional rubric covering methodological rigor, data preparation quality, analytical validity, and educational relevance. Based on this assessment, the Critic issues one of three verdicts: pass, revise, or abort. A “revise” verdict triggers a top‑down revision cascade: the orchestrator identifies the lowest‑level agent implicated by the Critic’s feedback and re‑executes that agent and all downstream agents, respecting a configurable maximum number of revision cycles (default two). If the maximum is exhausted without a pass, the pipeline proceeds to writing with an “unverified” flag. An “abort” verdict halts the run and logs the failure.

The Writer agent fills a predefined ACM sigconf LaTeX template with placeholders for sections, tables, figures, and a BibTeX bibliography generated from real Semantic Scholar metadata, producing a ready‑to‑submit manuscript. By separating concerns—LLMs handle prose generation, while code scaffolding, data validation, and LaTeX formatting are managed by deterministic components—the system mitigates common LLM hallucinations and formatting errors.

The authors compare EDM‑ARS to prior general‑purpose systems, highlighting three distinguishing design principles: (1) structural encoding of domain knowledge via the data registry and validation rules, (2) explicit separation of LLM‑generated prose from deterministic code and LaTeX scaffolding, and (3) custom orchestration rather than reliance on third‑party frameworks (e.g., LangChain), which affords precise control over state management, checkpointing, and revision routing.

Evaluation on the HSLS:09 dataset demonstrates that EDM‑ARS can generate complete manuscripts with accurate citations, validated predictive analyses, and fairness diagnostics comparable to manually authored papers. The checkpoint and error‑taxonomy mechanisms significantly reduce wasted computation by catching failures early (e.g., data validation errors, JSON parsing issues, insufficient sample size). Limitations include the current single‑dataset focus, formulaic writing style, and the absence of causal inference, transfer learning, and psychometric modeling capabilities. The roadmap outlines phased extensions to support quasi‑experimental designs, heterogeneous treatment effect estimation, cross‑domain transfer learning, and multi‑dataset generalization.

Finally, the system is released as open‑source software with detailed documentation, encouraging the EDM community to adopt, extend, and contribute to the platform. EDM‑ARS represents a concrete step toward fully autonomous, domain‑aware scientific research pipelines in education, promising to accelerate discovery while maintaining methodological rigor.