Exceptional Behaviors: How Frequently Are They Tested?

Exceptions allow developers to handle error cases expected to occur infrequently. Ideally, good test suites should test both normal and exceptional behaviors to catch more bugs and avoid regressions. While current research analyzes exceptions that propagate to tests, it does not explore other exceptions that do not reach the tests. In this paper, we provide an empirical study to explore how frequently exceptional behaviors are tested in real-world systems. We consider both exceptions that propagate to tests and the ones that do not reach the tests. For this purpose, we run an instrumented version of test suites, monitor their execution, and collect information about the exceptions raised at runtime. We analyze the test suites of 25 Python systems, covering 5,372 executed methods, 17.9M calls, and 1.4M raised exceptions. We find that 21.4% of the executed methods do raise exceptions at runtime. In methods that raise exceptions, on the median, 1 in 10 calls exercise exceptional behaviors. Close to 80% of the methods that raise exceptions do so infrequently, but about 20% raise exceptions more frequently. Finally, we provide implications for researchers and practitioners. We suggest developing novel tools to support exercising exceptional behaviors and refactoring expensive try/except blocks. We also call attention to the fact that exception-raising behaviors are not necessarily “abnormal” or rare.

💡 Research Summary

The paper presents an empirical investigation of how frequently exceptional behaviors are exercised during the execution of test suites in real‑world Python projects. While prior work has focused on exceptions that propagate to test code (i.e., tests that explicitly assert that an exception is raised), this study also captures exceptions that are caught locally by try/except blocks and therefore never reach the test harness.

Methodology
The authors selected 25 widely‑used Python systems, comprising both popular applications (e.g., Flask, Requests, Jupyter Client) and standard‑library modules (e.g., calendar, pathlib). They instrumented the test runs with SpotFlow, a dynamic tracing tool built on Python’s sys.settrace facility. SpotFlow records, for every method executed during testing, the number of calls, the number of distinct execution paths, and any exceptions raised (both propagated and locally handled). In total, 5,372 application methods were executed, generating 17.9 million calls and 1.4 million raised exceptions. The dataset is publicly available.

Research Questions

RQ1 – How many methods raise exceptions at runtime?

• 21.4 % of the executed methods (1,150 out of 5,372) raise at least one exception.
• Exception‑raising methods receive on average four times more calls and execute three times more distinct paths than exception‑free methods (statistically significant with Mann‑Whitney p < 0.05).
• 200 distinct exception types were observed; the most common are generic built‑in exceptions (ValueError, TypeError, KeyError, etc.), while many exceptions appear in only a single method.

RQ2 – How frequently do calls on exception‑raising methods actually lead to exceptions?

• For the 1,150 exception‑raising methods, the median number of exception‑raising calls is 4, which corresponds to 10 % of all calls to those methods. In other words, on average one out of ten invocations exercises an exceptional path.
• The authors categorize methods by the proportion of exception‑raising calls:
  • Rare (≤ 10 %): 50 % of methods (576).
  • Occasional (> 10 % ≤ 50 %): 28.4 % (327).
  • Common (> 50 % < 90 %): 9.6 % (111).
  • Almost always (≥ 90 %): 11.8 % (136).
• Thus, while roughly 80 % of exception‑raising methods trigger exceptions infrequently, about 20 % do so relatively often, with a non‑trivial minority (≈12 %) raising exceptions in the majority of their calls.

RQ3 – How do exception‑raising methods and calls vary by system?

• 22 of the 25 projects contain more exception‑free than exception‑raising methods.
• 19 projects have a median proportion of exception‑raising calls per method below 30 %. This suggests that most systems have a modest overall exposure to exceptional behavior during testing.

Implications

1. Tool Support for Hidden Exceptions – Because many exceptions are caught locally and never surface in test assertions, developers lack visibility into whether those exceptional paths have been exercised. Tools that automatically surface “silent” exception occurrences during test runs could help developers write explicit exceptional tests (e.g., adding assertRaises or similar checks) and improve fault detection.

2. Refactoring Expensive Try/Except Blocks – The frequency analysis reveals that some try/except constructs are executed very often (up to 98 % of calls). Since exception handling in Python incurs runtime overhead, especially when exceptions are raised frequently, there is an opportunity for automated refactoring tools to suggest alternative designs (e.g., guard clauses, explicit checks) for high‑frequency exception‑raising code.

3. Rethinking “Abnormal” Exceptions – The finding that a substantial fraction of methods raise exceptions regularly challenges the common assumption that exceptions always represent rare, abnormal conditions. Researchers working on exception‑testing techniques should treat exceptions as a spectrum of behaviors rather than a binary “normal vs. abnormal” classification.

Conclusion
By instrumenting and analyzing the execution of test suites across a diverse set of Python projects, the authors provide the first large‑scale quantitative picture of how often exceptions are raised, how often they are exercised, and how these patterns differ across systems. The study demonstrates that while most exception‑raising code is exercised rarely, a non‑negligible portion is exercised frequently, and many exceptions remain invisible to developers because they are caught locally. These insights motivate the development of new testing tools, refactoring aids, and a more nuanced view of exception handling in software engineering research and practice.