Quality Gatekeepers: Investigating the Effects ofCode Review Bots on Pull Request Activities

Software bots have been facilitating several development activities in Open Source Software (OSS) projects, including code review. However, these bots may bring unexpected impacts to group dynamics, as frequently occurs with new technology adoption. Understanding and anticipating such effects is important for planning and management. To analyze these effects, we investigate how several activity indicators change after the adoption of a code review bot. We employed a regression discontinuity design on 1,194 software projects from GitHub. We also interviewed 12 practitioners, including open-source maintainers and contributors. Our results indicate that the adoption of code review bots increases the number of monthly merged pull requests, decreases monthly non-merged pull requests, and decreases communication among developers. From the developers’ perspective, these effects are explained by the transparency and confidence the bot comments introduce, in addition to the changes in the discussion focused on pull requests. Practitioners and maintainers may leverage our results to understand, or even predict, bot effects on their projects.

💡 Research Summary

This paper investigates how the adoption of code review bots influences pull‑request (PR) dynamics in open‑source software (OSS) projects hosted on GitHub. The authors combine a large‑scale quantitative analysis with a qualitative interview study to capture both statistical effects and developers’ perceptions.

Exploratory case study
Two well‑known OSS projects, Julia and CakePHP, were examined for one year before and after the introduction of the Codecov bot. Metrics collected included monthly counts of merged and non‑merged PRs, median number of comments (excluding bot comments), median time to merge or close a PR, and median number of commits per PR. Non‑parametric Mann‑Whitney‑Wilcoxon tests and Cliff’s Δ revealed statistically significant changes for all four indicators in at least one of the projects, prompting the formulation of four hypotheses (H1–H4) concerning increases in merged PRs, decreases in non‑merged PRs, changes in comment volume, and shifts in latency and commit activity.

Main quantitative study
To test the hypotheses on a broader basis, the authors extracted 1,194 GitHub repositories that adopted a code‑review bot at a known point in time. Using a Regression Discontinuity Design (RDD), the bot adoption date served as the cutoff, and monthly aggregates were compared for the 12 months before and after the cutoff. RDD is appropriate because it treats the adoption as an exogenous shock, allowing causal inference about the bot’s impact. The results are as follows:

• Merged PRs – a statistically significant increase (average effect ≈ 0.8 standard deviations, p < 0.001). Projects accepted more contributions after bot adoption.
• Non‑merged PRs – a significant decrease (≈ 0.9 σ, p < 0.001). The bot appears to filter low‑quality or incomplete PRs early, reducing the backlog of rejected submissions.
• Comment volume – overall reduction in human‑generated comments, indicating that the bot’s automated feedback replaces many routine questions and shifts discussions toward substantive code changes.
• Time‑to‑merge – modestly longer after adoption, reflecting additional verification steps (e.g., coverage reports, static analysis) performed by the bot. Time‑to‑close for rejected PRs showed mixed results across projects.

Qualitative interview study
To interpret the quantitative findings, the authors conducted semi‑structured interviews with 12 OSS practitioners (maintainers and contributors) who have experience with code‑review bots. Participants highlighted several themes:

• Transparency and confidence – Bot comments provide concrete metrics (coverage percentages, style violations) that increase maintainers’ confidence in merging PRs quickly.
• Shift in communication – Automated bot feedback reduces the need for back‑and‑forth human comments, focusing discussions on actual code modifications rather than clarification requests.
• Potential downsides – Overly strict or opaque bot messages can demotivate contributors, especially newcomers, leading to possible drop‑out.
• Process efficiency – The bot standardizes the review pipeline, lowering maintainer workload and making the triage of incoming PRs more predictable.

Implications
The study offers actionable insights for OSS project leaders: (1) anticipate higher merge rates and lower rejection volumes when planning bot adoption; (2) design bot feedback to be clear, transparent, and educational to preserve contributor motivation; (3) consider the trade‑off between added verification time and overall throughput; and (4) use RDD as a robust methodological tool for evaluating future tooling interventions in software ecosystems.

Limitations and future work
The analysis is limited to GitHub repositories and primarily to the Codecov bot; results may differ for other platforms (e.g., GitLab) or other types of bots (static analysis, CI/CD). Long‑term effects on contributor retention, project growth, and ecosystem health remain open questions. Future research could compare multiple bot families, explore cross‑platform generalizability, and examine longitudinal impacts beyond the 12‑month window.

Conclusion
Code review bots positively affect pull‑request activity by increasing merged PRs, decreasing non‑merged PRs, and reducing human communication overhead, while introducing a modest increase in verification latency. The benefits stem from the bots’ transparent, metric‑driven feedback, which boosts maintainer confidence and streamlines the review process. However, careful design of bot messages is essential to avoid discouraging contributors. The combined quantitative‑qualitative approach demonstrates that RDD can effectively isolate the causal impact of tooling changes in OSS environments.