Using a "Study of Studies" to help statistics students assess research findings

One learning goal of the introductory statistics course is to develop the ability to make sense of research findings in published papers. The Atlantic magazine regularly publishes a feature called “Study of Studies” that summarizes multiple articles published in a particular domain. We describe a classroom activity to develop this capacity using the “Study of Studies.” In this activity, students read capsule summaries of twelve research papers related to restaurants and dining that was published in April 2015. The selected papers report on topics such as how seating arrangement, server posture, plate color and size, and the use of background music relate to revenue, ambiance, and perceived food quality. The students are assigned one of the twelve papers to read and critique as part of a small group. Their group critiques are shared with the class and the instructor. A pilot study was conducted during the 2015-2016 academic year at Amherst College. Students noted that key details were not included in the published summary. They were generally skeptical of the published conclusions. The students often provided additional summarization of information from the journal articles that better describe the results. By independently assessing and comparing the original study conclusions with the capsule summary in the “Study of Studies,” students can practice developing judgment and assessing the validity of statistical results.

💡 Research Summary

The paper presents a classroom activity designed to develop undergraduate statistics students’ ability to evaluate research findings and to understand study design. The activity leverages a “Study of Studies” column from The Atlantic, specifically the April 2015 article “Diner Beware: How restaurants trick you into eating less and spending more” by Bourree Lam. That article synthesizes twelve peer‑reviewed research papers on restaurant‑related variables such as seating arrangement, server posture, plate color and size, and background music.

Implementation began by distributing the one‑page Atlantic summary to the whole class and reading it aloud. Students were then divided into groups of two to four and each group received one of the twelve original research articles. The articles varied in length from four to twenty‑eight pages (average ten pages). Within their groups, students skimmed the assigned paper, identified the study’s sampling design, determined whether the study was experimental (randomized) or observational, extracted the primary hypothesis and key results, and noted any limitations reported by the authors.

Students documented their analysis in a set of slides created with RMarkdown, a reproducible reporting tool that integrates code, narrative, and graphics. The slides were uploaded to RPubs (or otherwise shared with the instructor) and each group presented their findings for five to ten minutes. The presentation required students to compare the original article’s details with Lam’s terse summary, highlighting discrepancies, omissions, or over‑generalizations.

A pilot was conducted during the 2015‑2016 academic year at Amherst College with introductory and intermediate statistics courses (approximately 20–25 students per class, 80 minutes total). The Institutional Review Board approved the study. Results showed that students reliably identified core design elements of the original studies: sample size, research question, classification as experimental or observational, and the main hypothesis. For example, in Guéguen & Petr (2006) students noted that the experiment was conducted in a single small pizzeria in Brittany, France, with 88 patrons over three Saturday evenings, and they correctly recognized the cultural and geographic bias inherent in such a limited sample. They also captured the authors’ own acknowledgment that the small sample and single‑site design constrained external validity.

When evaluating Kimes & Robson (2004), students recognized the single‑blind design, the large sample of 1,413 diners, and the focus on table type and location. They critiqued Lam’s summary for implying a causal relationship (“banquette diners stayed the longest”) despite the original study’s observational nature and its exclusion of certain seating areas (e.g., bar and patio) and non‑peak hours. Students repeatedly pointed out that Lam’s column, by design, condenses complex findings into provocative sound bites, which can obscure methodological nuances such as randomization, control conditions, or statistical significance testing.

Overall, the activity succeeded in (1) reinforcing study‑design concepts taught in lectures, (2) fostering critical appraisal skills by juxtaposing a popular‑media synthesis with the primary literature, and (3) enhancing digital literacy through RMarkdown and web‑based sharing. The authors caution, however, that students may develop an overly skeptical stance (“knee‑jerk skepticism”) if instructors do not explicitly discuss the purpose of media summaries and the legitimate trade‑off between brevity and completeness.

The discussion suggests that similar activities could be built around other “Study of Studies” articles (e.g., “Gullible Brains,” “The Science of Beer Goggles,” “CEOs Behaving Badly”), allowing instructors to align topics with student interests. By situating statistical reasoning within real‑world contexts that students encounter in everyday media, the approach bridges the gap between abstract textbook examples and authentic research evaluation.

In conclusion, the paper provides empirical evidence that a short, structured classroom exercise—reading a popular‑media summary, dissecting the original study, and presenting a comparative critique—effectively cultivates undergraduate statistics students’ competence in assessing research validity, understanding design limitations, and communicating statistical findings. Future work could examine longitudinal retention of these skills and explore adaptation to other disciplinary domains.