Frequency Distribution of Error Messages

Which programming error messages are the most common? We investigate this question, motivated by writing error explanations for novices. We consider large data sets in Python and Java that include both syntax and run-time errors. In both data sets, after grouping essentially identical messages, the error message frequencies empirically resemble Zipf-Mandelbrot distributions. We use a maximum-likelihood approach to fit the distribution parameters. This gives one possible way to contrast languages or compilers quantitatively.

💡 Research Summary

The paper investigates which programming error messages occur most frequently, motivated by the need to provide novice‑friendly explanations in educational environments. The authors collected massive error logs from two beginner‑oriented platforms: CS Circles (Python) and BlueJ’s Blackbox project (Java). The Python corpus contains about 640 000 error instances out of 1.6 million submissions, while the Java corpus includes roughly 4 million compile‑time errors and 130 000 run‑time errors from millions of compile and invoke events.

Because raw error strings embed user‑specific identifiers (variable names, file paths, line numbers), the authors designed a sanitization pipeline using regular‑expression rules (≈20 for Python, ≈50 for Java) to collapse semantically identical messages. After sanitization, they identified 283 distinct Python error types and 572 distinct Java error types. Frequency counts (Fk) were then ordered by rank (k = 1 for the most common error).

Initial log‑log plots of Fk versus k revealed that a simple discrete power‑law (Fk ∝ k⁻ᵞ) does not fit the data; the points curve away from a straight line. The authors therefore turned to the Zipf‑Mandelbrot family, defined by Fk ∝ (k + t)⁻ᵞ, where t is a shift parameter. By manually adjusting t (≈60 for Python, ≈70 for Java) the transformed data align closely with a straight line, suggesting a good fit.

To estimate the parameters rigorously, the authors employed a maximum‑likelihood approach, yielding γ≈6 for both languages and t values consistent with the visual fit. The exponent γ quantifies the “steepness” of the distribution: a larger γ means a few error messages dominate the total error count, while a smaller γ indicates a flatter distribution with many different errors occurring. Consequently, γ can serve as a quantitative metric for comparing the error‑friendliness of languages or compilers.

The paper offers a plausibility argument: starting from a pure power‑law distribution of fine‑grained error messages, merging several messages into a single higher‑level one (as occurs during sanitization) creates an outlier and shifts the remaining points leftward, producing a Zipf‑Mandelbrot shape. This “message‑merging” hypothesis explains why the observed data deviate from a pure power law.

Beyond statistical modeling, the authors discuss practical implications. They authored beginner‑oriented explanations for the 36 most frequent errors and implemented a system that attaches a clickable pop‑up to the original compiler/runtime message, delivering the elaboration on demand. This lightweight augmentation approach is presented as a scalable solution for any beginner‑focused programming environment.

The work is positioned relative to prior literature on error‑message design, manual error categorization, and large‑scale error‑log analysis. While many studies focus on qualitative categorization or on improving compiler diagnostics, this paper is, to the authors’ knowledge, the first to link error‑message frequencies to a Zipf‑Mandelbrot distribution. The authors caution that a single numeric metric should not dominate usability assessments; entropy, time‑to‑fix, and learner outcomes remain essential complementary measures.

In conclusion, the study demonstrates that error‑message frequencies in large novice datasets follow a Zipf‑Mandelbrot distribution, provides a principled method for estimating its parameters, and shows how these insights can guide the design of effective, data‑driven error‑explanation tools. Future work is suggested to extend the analysis to other languages, professional developer populations, and to integrate dynamic feedback loops that adapt explanations based on real‑time learner interaction.