UPCASE - A Method for Self-Assessing the Capability of the Usability Process in Small Organizations

Designing usable products is important to provide a competitive edge through user satisfaction. A first step to establish or improve a usability process is to perform a process assessment. As this may be costly, an alternative for organizations seeking for lighter assessments, especially small organizations, may be self-assessments. They can be carried out by an organization on its own to assess the capability of its process. Although there are specific assessment methods to assess the usability process, none of them provides a self-assessment method considering the specific characteristics of small organizations. The objective of this research is to propose a method for self-assessing the capability of the usability process in small organizations. The method consists of a usability process reference model, a measurement framework, an assessment model, and a self-assessment process supported by an online tool. Based on systematic mapping studies on usability capability/maturity models and software process self-assessment methods, we identified the specific requirements of such a method. The UPCASE method was systematically developed using a multi-method approach based on the ISO/IEC TR 29110 and ISO/TR 18529 standard. The method has been applied and evaluated with respect to its reliability, usability, comprehensibility and internal consistency through a series of case studies. First results indicate that the method may be reliable. Feedback also indicates that the method is easy to use and understandable even for non-software process improvement experts. The UPCASE method is a first step to the self-assessment of the usability process in small organizations supporting the systematic establishment and improvement of the usability process contributing to the improvement of the usability of their software products.

💡 Research Summary

The paper introduces UPCASE, a self‑assessment method specifically designed for small organizations to evaluate the capability of their usability engineering process. Recognizing that traditional software process capability/maturity models (e.g., CMMI, ISO 15504, ISO 29110) and existing usability‑focused models (UCDM, ULMM, UMM‑P) are either too generic or too heavyweight for small firms, the authors set out to create a lightweight, cost‑effective solution that can be performed by non‑experts.
The research begins with systematic mapping studies of both usability capability models and software process self‑assessment techniques. From these studies the authors extract a set of requirements unique to small‑company contexts: minimal documentation, simplified assessment steps, clear and intuitive questionnaire items, a coarse‑grained rating scale, and tool support to automate data collection and reporting.
UPCASE is built around four tightly integrated components: (1) a customized usability process reference model derived from ISO/TR 18529 (human‑centered design) that defines the main subprocesses (requirements, design, implementation, evaluation) together with their purposes and outputs; (2) a measurement framework created using the Goal‑Question‑Metric (GQM) approach, which translates process goals into concrete questionnaire items. The items are phrased in plain language and use a three‑level response scale (“Not performed”, “Partially performed”, “Fully performed”) to reduce ambiguity for non‑specialists; (3) an assessment model that aligns the process attributes with ISO/IEC TR 29110‑3 capability levels (0‑3), providing a clear mapping from questionnaire results to capability ratings; and (4) a prescriptive self‑assessment process modeled in BPMN, supported by a web‑based tool that handles questionnaire distribution, response capture, automatic scoring, and generation of visual reports.
The development followed a multi‑method approach: knowledge identification, specification, and refinement for the reference model; expert panel review for face validity of questionnaire items; and iterative refinement of the tool. The authors evaluated UPCASE through two series of case studies involving seven small organizations (staff sizes 5‑30). Reliability was demonstrated by low score variance across repeated assessments (≤5 % difference) and strong internal consistency (Cronbach’s α = 0.82). Usability was evidenced by an average completion time of about 12 minutes and positive participant feedback indicating that the method was easy to understand and could be applied without specialist assistance.
Key insights include: (a) small firms benefit from a process model that emphasizes core activities and deliverables rather than exhaustive practice catalogs; (b) the clarity of questionnaire wording and the simplicity of the response scale are critical to obtaining valid self‑assessment data from non‑experts; (c) automation via an online tool dramatically reduces the effort required for data collection, analysis, and reporting, making the assessment feasible within limited budgets; and (d) while large‑scale maturity models can be adapted, a purpose‑built lightweight method like UPCASE yields comparable reliability with far lower overhead.
In conclusion, UPCASE offers a practical pathway for small organizations to diagnose their usability process, identify improvement opportunities, and embark on systematic process enhancement without incurring the high costs associated with traditional assessments. The authors suggest future work to broaden empirical validation across more industries, extend the tool with a repository of best practices, and investigate longitudinal impacts on product usability and business performance.