Experimental designs for multiple-level responses, with application to a large-scale educational intervention

Educational research often studies subjects that are in naturally clustered groups of classrooms or schools. When designing a randomized experiment to evaluate an intervention directed at teachers, but with effects on teachers and their students, the power or anticipated variance for the treatment effect needs to be examined at both levels. If the treatment is applied to clusters, power is usually reduced. At the same time, a cluster design decreases the probability of contamination, and contamination can also reduce power to detect a treatment effect. Designs that are optimal at one level may be inefficient for estimating the treatment effect at another level. In this paper we study the efficiency of three designs and their ability to detect a treatment effect: randomize schools to treatment, randomize teachers within schools to treatment, and completely randomize teachers to treatment. The three designs are compared for both the teacher and student level within the mixed model framework, and a simulation study is conducted to compare expected treatment variances for the three designs with various levels of correlation within and between clusters. We present a computer program that study designers can use to explore the anticipated variances of treatment effects under proposed experimental designs and settings.

💡 Research Summary

The paper tackles a common dilemma in educational research: how to design a randomized experiment when an intervention is delivered to teachers but its effects are measured both at the teacher level and the student level. Because teachers and students are naturally clustered within classrooms and schools, the choice of randomization unit has profound implications for statistical power, bias, and the risk of contamination (i.e., spill‑over of the treatment to control units). The authors compare three plausible designs: (A) randomizing entire schools, (B) randomizing teachers within schools (a blocked design), and (C) completely randomizing teachers irrespective of school affiliation.

Using a linear mixed‑effects framework, they derive closed‑form expressions for the variance of the estimated treatment effect under each design. The model includes random effects for schools, teachers, and students, allowing the intraclass correlations (ICCs) at the school‑teacher level (ρSC) and the teacher‑student level (ρTS) to be explicitly represented. The variance formulas reveal how the number of schools, teachers per school, and students per teacher interact with the ICCs to determine efficiency.

A comprehensive simulation study explores a wide range of realistic parameter values: school counts from 30 to 100, teachers per school from 5 to 10, students per teacher from 20 to 30, and ICCs ranging from 0.01 to 0.15 (school‑teacher) and 0.05 to 0.30 (teacher‑student). Two contamination scenarios are examined – none (contamination effect = 0) and modest (contamination effect = 0.05). Results show that school‑level randomization (Design A) is most robust when the school‑teacher ICC is high or when any contamination is present, because it eliminates within‑school spill‑over. However, it suffers a loss of power when the ICC is low and the number of schools is limited. Teacher‑within‑school randomization (Design B) yields the smallest variance when the teacher‑student ICC is large and contamination is negligible, exploiting within‑school variation while still controlling for school effects. Complete teacher randomization (Design C) can be the most efficient in a perfectly isolated setting, but its variance inflates dramatically once contamination is introduced, making it the least reliable in practice.

To translate these theoretical findings into actionable guidance, the authors provide an R‑based software tool (named “multilevelDesign”). Users input anticipated cluster sizes, ICC values, total sample size, and a hypothesized contamination level; the program outputs expected treatment‑effect variances, statistical power, and required sample sizes for each design. The tool also produces visualizations that help researchers weigh trade‑offs between cost (e.g., number of schools to recruit) and statistical efficiency.

The key take‑away is that no single design dominates across all scenarios. If the primary interest lies in teacher outcomes and the risk of contamination is judged low, randomizing teachers within schools (Design B) offers a cost‑effective solution. If the focus is on student outcomes, or if there is any concern that teachers might share intervention materials across control groups, randomizing at the school level (Design A) provides a safer, albeit sometimes less powerful, alternative. The authors stress that researchers should conduct a pre‑study power analysis using realistic ICC estimates and contamination assumptions, and they argue that the provided software makes this process accessible.

Overall, the paper contributes a rigorous comparative analysis of multi‑level experimental designs, clarifies how intra‑cluster correlations and contamination interact to affect efficiency, and equips practitioners with a practical computational resource for designing robust educational interventions.