Assessing the association between pre-course metrics of student preparation and student performance in introductory statistics: Results from early data on simulation-based inference vs. nonsimulation based inference

The recent simulation-based inference (SBI) movement in algebra-based introductory statistics courses (Stat 101) has provided preliminary evidence of improved student conceptual understanding and retention. However, little is known about whether these positive effects are preferentially distributed across types of students entering the course. We consider how two metrics of Stat 101 student preparation (pre-course performance on concept inventory and math ACT score) may or may not be associated with end of course student performance on conceptual inventories. Students across all preparation levels tended to show improvement in Stat 101, but more improvement was observed across all student preparation levels in early versions of a SBI course. Furthermore, students’ gains tended to be similar regardless of whether students entered the course with more preparation or less. Recent data on a sample of students using a current version of an SBI course showed similar results, though direct comparison with non-SBI students was not possible. Overall, our analysis provides additional evidence that SBI curricula are effective at improving students’ conceptual understanding of statistical ideas post-course regardless student preparation. Further work is needed to better understand nuances of student improvement based on other student demographics, prior coursework, as well as instructor and institutional variables.

💡 Research Summary

The paper investigates whether the benefits of simulation‑based inference (SBI) curricula in introductory algebra‑based statistics courses (Stat 101) are distributed evenly across students with different levels of prior preparation. Two preparation metrics are examined: (1) performance on a pre‑course conceptual inventory (the CAOS pre‑test) and (2) math ACT scores. The authors compare three instructional conditions: (a) a traditional “consensus” curriculum using a standard textbook, (b) an early version of an SBI curriculum, and (c) a newer, revised SBI curriculum.

Data were collected from three samples. The first two samples come from the same two small liberal‑arts colleges. The consensus group includes 289 students (2007 and spring 2011) and the early‑SBI group includes 366 students (2009 and 2011‑12). All students completed the CAOS test at the beginning and end of the semester, allowing a direct measurement of change in conceptual understanding. The third sample consists of 1,078 students from 13 institutions (including community colleges, private universities, AP high‑school courses, and large public universities) who used a modified CAOS instrument in the 2013‑14 academic year with the newer SBI curriculum.

Students were stratified into roughly equal tertiles based on (i) pre‑test CAOS scores (low ≤40 % correct, middle 40‑50 %, high ≥50 %) and (ii) ACT math scores (low ≤22, middle 23‑26, high ≥27). Change scores (post‑test minus pre‑test) were analyzed with paired t‑tests and linear models that included curriculum and institution as predictors. Subscale analyses examined nine CAOS content areas (e.g., data collection, probability, tests of significance).

Key findings:

1. Overall gains – Both curricula produced significant pre‑to‑post gains (p < 0.001). The early‑SBI group showed a larger average gain (11.0 % points) than the consensus group (7.8 % points), a difference of 3.2 % points after adjusting for institution.

2. Performance‑level effects – When stratified by pre‑test performance, early‑SBI students outperformed consensus students in every tertile, with gains 2–3 % points higher. The middle‑performing group’s advantage (3.1 % points, SE = 1.5, p < 0.05) was statistically significant; low and high groups showed non‑significant trends in the same direction.

3. ACT‑score effects – Across ACT tertiles, early‑SBI students achieved higher gains than consensus students in all three groups. The low ACT group showed the largest curriculum effect (8.2 % points higher gain, p < 0.001). An ANOVA comparing models with and without an ACT‑by‑curriculum interaction was non‑significant (p = 0.15), indicating that the curriculum advantage was fairly consistent across ACT levels.

4. Subscale improvements – For students with low pre‑test scores, early‑SBI produced significant gains on three subscales: data collection & design (+9.4 % points), tests of significance (+8.4 % points), and probability (+15.8 % points). Similar patterns, though sometimes not statistically significant, were observed for middle‑ and high‑performing students.

5. Newer SBI curriculum – The large multi‑institution sample using the revised SBI curriculum also showed overall pre‑to‑post improvement, confirming that the benefits of SBI persist in a more mature implementation. However, because no contemporaneous consensus control was collected, direct effect size comparisons are not possible.

The authors argue that these results address three limitations of prior work: (a) they focus on change in validated conceptual understanding rather than grades, (b) they consider how gains vary with prior ability, and (c) they compare curricula directly. Limitations include the reliance on ACT data from a single institution, the lack of a control group for the newer SBI curriculum, and the absence of detailed demographic controls (e.g., gender, major, high‑school coursework).

Practical implications: SBI curricula appear to raise conceptual understanding for all students, regardless of prior math ability, and may be especially beneficial for those with weaker mathematical backgrounds. Instructors can therefore adopt SBI without needing to differentiate instruction based on incoming preparation. Future research should incorporate a broader range of institutions, control for instructor effects, and examine long‑term retention and transfer to real‑world data analysis tasks.

In summary, the study provides robust evidence that simulation‑based inference instruction improves students’ statistical reasoning across the spectrum of preparation, supporting wider adoption of SBI approaches in introductory statistics education.