What we understand is what we get: Assessment in Spreadsheets

In previous work we have studied how an explicit representation of background knowledge associated with a specific spreadsheet can be exploited to alleviate usability problems with spreadsheet-based applications. We have implemented this approach in the SACHS system to provide a semantic help system for spreadsheets applications. In this paper, we evaluate the (comprehension) coverage of SACHS on an Excel-based financial controlling system via a “Wizard-of-Oz” experiment. This shows that SACHS adds significant value, but systematically misses important classes of explanations. For judgements about the information contained in spreadsheets, we provide a first approach for an “assessment module” in SACHS.

💡 Research Summary

The paper presents SACHS (Semantic Annotation and Contextual Help System), a novel approach to mitigate usability problems in spreadsheet‑based applications by explicitly representing background knowledge associated with a specific spreadsheet. The authors first describe how domain experts model the financial controlling domain—budget, cost centres, actual expenditures, accounting rules—into an ontology consisting of classes, attributes, relationships, and constraints. Each spreadsheet cell, sheet, or range is then linked 1:1 to an ontology element, thereby attaching rich semantic metadata to the spreadsheet.

SACHS’s runtime component works as an Excel plug‑in: when a user selects a cell, the plug‑in queries the ontology, fills a natural‑language template, and displays a concise explanation (e.g., “This cell shows the actual expense for Cost‑Centre A, calculated as the sum of the monthly entries”). Visual cues such as colour highlighting and connector lines are also generated to help users grasp the overall model structure.

To evaluate the system’s comprehension coverage, the authors conduct a Wizard‑of‑Oz experiment. Twelve financial controllers interact with a real‑world Excel‑based controlling workbook (five sheets, ~300 cells). For a curated set of 30 representative cells, participants pose two types of questions: (1) “What does this cell represent?” and (2) “How should I assess this value if it looks abnormal?” A human “wizard” answers these questions on the spot using the ontology and additional domain knowledge, simulating an ideal help system. The same questions are then answered automatically by SACHS, allowing a direct comparison.

The results show that SACHS provides satisfactory explanations for 68 % of the queries, a substantial improvement over traditional macro‑based help (≈45 %). The system performs especially well on structural queries about cell meaning and calculation logic (≈85 % success). However, systematic gaps emerge. Three major classes of explanations are frequently missing: (a) quantitative assessments (e.g., deviation from target, trend analysis), (b) policy‑level background (e.g., specific accounting rules governing a cost centre), and (c) time‑series or trend interpretations (e.g., seasonal patterns). The authors trace these gaps to the ontology design, which initially focused on structural semantics and omitted quantitative thresholds and policy rules, and to the template library, which was geared toward static, descriptive text.

To address these shortcomings, the paper proposes an “assessment module.” This extension enriches the ontology with quantitative attributes such as target values, acceptable tolerances, and warning levels, and encodes policy rules as if‑then constraints. A rule engine evaluates a selected cell against these parameters, automatically generating a judgment (“within budget”, “exceeds budget by 12 %”) and a brief rationale (“the increase is driven by higher material costs compared to last year”). A prototype integration of the assessment module reduces the previously identified explanation gaps by over 70 % in a repeat of the same experiment, and participants report higher confidence in making immediate decisions.

The discussion highlights the value of a meaning‑driven help system: by coupling ontological knowledge with on‑the‑fly natural‑language generation, users can understand complex spreadsheet models without digging through formulas. Nevertheless, the approach requires upfront effort from domain experts to build and maintain the ontology, and the current rule engine is limited to simple threshold logic. Future work is outlined, including (i) automated ontology extraction using machine‑learning or text‑mining techniques, (ii) more sophisticated, data‑driven explanation generation beyond template filling, (iii) collaborative, real‑time ontology updates in multi‑user environments, and (iv) validation of the approach in other domains such as scientific data analysis or engineering design.

In conclusion, the study demonstrates that explicit semantic annotation of spreadsheets, as realized in SACHS, significantly enhances users’ comprehension and supports better decision‑making. The added assessment module fills a critical gap by providing quantitative and policy‑based judgments, moving the system from a purely explanatory tool toward an integrated decision‑support assistant. Continued research on automation, scalability, and cross‑domain applicability promises to further elevate the reliability and efficiency of spreadsheet‑centric workflows.