Using and Designing Platforms for In Vivo Education Experiments

In contrast to typical laboratory experiments, the everyday use of online educational resources by large populations and the prevalence of software infrastructure for A/B testing leads us to consider how platforms can embed in vivo experiments that do not merely support research, but ensure practical improvements to their educational components. Examples are presented of randomized experimental comparisons conducted by subsets of the authors in three widely used online educational platforms Khan Academy, edX, and ASSISTments. We suggest design principles for platform technology to support randomized experiments that lead to practical improvements enabling Iterative Improvement and Collaborative Work and explain the benefit of their implementation by WPI co-authors in the ASSISTments platform.

💡 Research Summary

The paper introduces a paradigm called “in‑vivo education experiments,” which leverages the massive, everyday usage of online learning platforms to run randomized controlled trials directly within the learning environment. Unlike traditional laboratory‑based studies that rely on small, artificial samples, the authors argue that platforms such as Khan Academy, edX, and ASSISTments already possess the technical infrastructure (e.g., A/B testing pipelines, user tracking, and data analytics) needed to embed rigorous experiments at scale. By doing so, researchers and educators can obtain real‑world evidence about instructional designs, instantly apply successful interventions, and iterate rapidly toward better learning outcomes.

Three concrete case studies illustrate how this approach works in practice. At Khan Academy, the authors randomized video playback speed and problem difficulty for subsets of learners, finding that modestly increased speeds did not harm comprehension while adaptive difficulty modestly extended engagement time. On edX, two alternative course navigation structures—linear versus learner‑chosen non‑linear pathways—were compared; the non‑linear design fostered greater self‑directed activity but did not produce a statistically significant difference in overall course grades. The most sophisticated example comes from the ASSISTments platform, where teachers can upload new problem types, feedback scripts, or hints without any programming. The system automatically assigns these variations to random learner groups, collects interaction logs in real time, and presents statistical results through an integrated dashboard. Two ASSISTments experiments—immediate feedback versus delayed feedback, and varying the timing of hint delivery—demonstrated measurable reductions in error rates and faster problem‑solving.

From these experiences the authors distill two design principles for platform‑level support of in‑vivo experiments. The first, “Iterative Improvement,” emphasizes a tight feedback loop: experiment outcomes are fed back into the product as quickly as possible, enabling the next hypothesis to be tested on a freshly updated interface. The second, “Collaborative Work,” calls for shared tools and data access among researchers, teachers, and developers so that experiment design, deployment, analysis, and interpretation become a collective activity rather than a siloed effort.

Technically, the paper proposes a modular architecture consisting of (1) an experiment‑parameter manager that centralizes variable definitions, version control, and random assignment logic; (2) a user‑segmentation engine that creates stratified groups based on demographics, prior knowledge, or behavioral signals; (3) a real‑time logging and analytics pipeline that streams interaction events, applies pre‑specified statistical tests, and flags significant effects; and (4) a visualization/feedback dashboard that displays key metrics (e.g., mastery, time on task, correctness) and offers guidance on statistical significance. These components are implemented as micro‑services and containerized deployments, ensuring scalability across millions of users while respecting privacy regulations through consent mechanisms and minimal data collection.

The discussion acknowledges both the promise and the challenges of large‑scale in‑vivo experimentation. While the statistical power is high, the complexity of experimental design can introduce bias or implementation errors if not carefully managed. Moreover, educators and learners may be unaware that they are part of an experiment, potentially affecting acceptance of results. To mitigate these issues, the authors recommend professional development for teachers on experimental thinking, as well as automated design templates and clear ethical guidelines embedded in the platform.

In conclusion, the authors argue that embedding randomized experiments into the core of online education platforms transforms these systems into living laboratories. This not only strengthens the empirical foundation of educational technology but also creates a sustainable cycle of evidence‑based improvement. The ASSISTments implementation serves as a concrete model that other platforms can emulate, suggesting a path toward a universal, collaborative framework for in‑vivo learning research.