In Search of a Taxonomy for Classifying Qualitative Spreadsheet Errors

Most organizations use large and complex spreadsheets that are embedded in their mission-critical processes and are used for decision-making purposes. Identification of the various types of errors that can be present in these spreadsheets is, therefore, an important control that organizations can use to govern their spreadsheets. In this paper, we propose a taxonomy for categorizing qualitative errors in spreadsheet models that offers a framework for evaluating the readiness of a spreadsheet model before it is released for use by others in the organization. The classification was developed based on types of qualitative errors identified in the literature and errors committed by end-users in developing a spreadsheet model for Panko’s (1996) “Wall problem”. Closer inspection of the errors reveals four logical groupings of the errors creating four categories of qualitative errors. The usability and limitations of the proposed taxonomy and areas for future extension are discussed.

💡 Research Summary

The paper addresses the growing concern that spreadsheets, which are often embedded in mission‑critical business processes, can contain not only quantitative errors (incorrect numbers or logic) but also qualitative errors—design and structural flaws that increase the likelihood of future quantitative mistakes. While the literature has produced several taxonomies for quantitative errors, there is a notable gap in systematic classifications for qualitative errors. The authors therefore set out to develop a taxonomy that can be used during the design and development phases of a spreadsheet to assess its readiness before release to operational users.

The authors begin by reviewing existing error classifications, notably Panko and Halverson’s early work (1996) and its later revision by Panko and Aurigemma (2010), which introduced a life‑cycle view distinguishing design‑time and operational errors. They also reference Rajalingham et al. (2000), who offered an umbrella taxonomy for qualitative errors but without sufficient granularity for practical testing. To build a more detailed framework, the authors adopt a two‑pronged approach. First, they synthesize qualitative error types reported in the literature and map them onto four high‑level categories: (1) Input Data Structure, (2) Semantics, (3) Extendibility & Formula Integrity, and (4) Readability & Copy‑Paste Issues. Second, they conduct an empirical study using 104 spreadsheet models created by participants to solve Panko’s “Wall problem.” By analysing the errors that actually appeared in these models, they discover that many of the literature‑derived categories are too coarse and that additional atomic error items are needed.

The resulting taxonomy records the presence (a binary “exists” flag) of each error type rather than counting every occurrence. This decision was motivated by the observation that counting instances proved inefficient and produced inconsistent results across reviewers. Because qualitative errors are latent—often hidden until a user performs a what‑if analysis—the authors argue that flagging any occurrence is sufficient to trigger a comprehensive review of the model. For example, a hard‑coded interest rate in a financial model may appear only once, but its presence signals the need to locate and replace all similar hard‑codings before the spreadsheet is deployed.

The four categories are illustrated with concrete examples from the Wall problem:

1. Input Data Structure – includes hard‑coding (or “jamming”) of input values into formulas, duplication of the same input across multiple cells, and failure to clearly identify input cells (e.g., lack of a dedicated input module or visual cues).

2. Semantics – covers missing or incorrect cell documentation, ambiguous labels, and formatting that can mislead users about the meaning of a value (e.g., profit margin mislabeled as total cost).

3. Extendibility & Formula Integrity – captures design flaws such as references to external worksheets, use of macros, array formulas, circular references, and reliance on functions like OFFSET or INDIRECT that make the model brittle when extended.

4. Readability & Copy‑Paste Issues – addresses problems that arise when users copy or move rows/columns: absolute vs. relative reference errors, formulas that break when pasted, and layout choices that hinder easy duplication of logic.

The taxonomy is positioned within the broader error classification framework (Figure 2 in the paper) as a subset of design‑time qualitative errors, distinct from execution errors that occur during operational use (e.g., user‑entered data entry mistakes). The authors argue that by applying this taxonomy during testing, auditors and spreadsheet reviewers gain a systematic rubric for assessing “usability” and “robustness” beyond mere numerical correctness.

Limitations are acknowledged. The study focuses exclusively on design‑time errors; it does not address user‑generated execution errors, which require different controls such as cell protection or data validation. Moreover, the empirical validation is limited to a single problem domain (the Wall problem), raising questions about generalizability across financial, engineering, or scientific spreadsheets. The taxonomy’s four categories, while comprehensive for the authors’ sample, may need refinement or expansion to capture domain‑specific nuances.

Future work suggested includes: (a) extending the taxonomy to cover operational‑phase qualitative errors; (b) testing the framework on a broader set of real‑world spreadsheets from diverse industries; (c) integrating the taxonomy with automated error‑detection tools to streamline the “exists” flagging process; and (d) investigating the causal relationship between specific qualitative errors and subsequent quantitative failures.

In conclusion, the paper contributes a practical, empirically‑grounded taxonomy for qualitative spreadsheet errors, filling a notable gap in the spreadsheet governance literature. By providing a clear set of error categories and a binary recording method, the taxonomy offers a usable checklist for spreadsheet testing, internal audit, and governance programs, thereby helping organizations ensure that numerically accurate spreadsheets are also structurally sound before they are put into production.