An Empirical Study of Bots in Software Development -- Characteristics and Challenges from a Practitioner's Perspective

💡 Research Summary

The paper presents a mixed‑method empirical investigation into software development bots (DevBots) from the perspective of practitioners. The authors note that modern software development involves many repetitive, non‑coding tasks (code review, bug triage, infrastructure maintenance, on‑call duties, etc.) that can distract developers and reduce productivity. As a response, a growing number of projects adopt automated tools called “bots” to offload such tedious work. However, prior research suffers from a lack of consensus on what constitutes a DevBot, how it differs from ordinary development tools, and what benefits or challenges it brings. Existing taxonomies are largely based on tool characteristics rather than user perceptions.

To fill this gap, the authors conducted a six‑month study comprising (1) semi‑structured interviews with 21 industry professionals and (2) a web‑based survey completed by 111 software developers and IT professionals (60 of whom reported using DevBots). The interview sample spans large corporations, SMEs, and a startup, with an average of 11 years of experience. Grounded Theory (Straussian variant) was used for open and axial coding, memoing, and constant comparison, leading to the identification of three distinct user personas:

• Charlie (Chat‑bot persona) – Views DevBots primarily as information‑integration tools accessed via natural‑language interfaces. Users expect easy retrieval of data and the ability to trigger simple maintenance tasks through chat.
• Alex (Autonomous‑bot persona) – Considers DevBots as agents that autonomously perform repetitive, low‑complexity tasks without explicit user prompts. The focus is on eliminating manual steps.
• Sam (Smart‑bot persona) – Defines a DevBot by its “smartness”: the capacity to handle non‑trivial, context‑aware tasks or generate insights that would be difficult for a human to obtain.

Survey analysis shows that among respondents who actually use bots, 48 % align with Charlie, 19 % with Alex, and 13 % with Sam; the remaining 20 % could not be mapped to any persona. Each persona also differs in the way it expects productivity gains: Charlie seeks quick access to information; Alex seeks automation of mundane chores; Sam seeks higher‑level assistance that can solve complex problems or synthesize data.

The study uncovers three major challenges. First, trust and reliability: autonomous and smart bots must have extremely low false‑positive rates and robust test suites to avoid unintended actions. Second, usability and predictability: chat‑bots that attempt rich natural‑language understanding are often perceived as unpredictable, while overly simplistic command‑based bots are seen as limited. Third, lack of general‑purpose smart bots: Sam‑style bots are typically bespoke, built internally by large organizations, leading to high development and maintenance costs and a scarcity of off‑the‑shelf solutions.

Related work is reviewed, highlighting earlier taxonomies (e.g., Lebeuf & Storey, Paikari & van der Hoek) that classify bots based on functional facets but do not capture user‑centric distinctions. The authors argue that their “DevBot vs. Plain Old Development Tool (PODT)” framing, grounded in practitioner viewpoints, complements existing classifications and provides a clearer lens for future research.

In conclusion, the paper delivers a practitioner‑driven definition of DevBots, identifies three archetypal user groups with distinct expectations, quantifies their prevalence, and outlines concrete challenges—particularly the need for trustworthy, low‑error autonomous behavior and the development of general‑purpose smart bots. These insights aim to guide both researchers (in formulating precise research questions and evaluation criteria) and industry practitioners (in selecting, designing, and deploying bots that align with their team’s needs and risk tolerance).