What do Support Analysts Know about Their Customers? On the Study and Prediction of Support Ticket Escalations in Large Software Organizations

Understanding and keeping the customer happy is a central tenet of requirements engineering. Strategies to gather, analyze, and negotiate requirements are complemented by efforts to manage customer input after products have been deployed. For the latter, support tickets are key in allowing customers to submit their issues, bug reports, and feature requests. Whenever insufficient attention is given to support issues, however, their escalation to management is time-consuming and expensive, especially for large organizations managing hundreds of customers and thousands of support tickets. Our work provides a step towards simplifying the job of support analysts and managers, particularly in predicting the risk of escalating support tickets. In a field study at our large industrial partner, IBM, we used a design science methodology to characterize the support process and data available to IBM analysts in managing escalations. Through iterative cycles of design and evaluation, we translated our understanding of support analysts’ expert knowledge of their customers into features of a support ticket model to be implemented into a Machine Learning model to predict support ticket escalations. We trained and evaluated our Machine Learning model on over 2.5 million support tickets and 10,000 escalations, obtaining a recall of 79.9% and an 80.8% reduction in the workload for support analysts looking to identify support tickets at risk of escalation. Further on-site evaluations, through a prototype tool we developed to implement our Machine Learning techniques in practice, showed more efficient weekly support-ticket-management meetings. The features we developed in the Support Ticket Model are designed to serve as a starting place for organizations interested in implementing our model to predict support ticket escalations, and for future researchers to build on to advance research in …

💡 Research Summary

The paper investigates how support analysts’ tacit knowledge about customers can be captured, formalized, and leveraged to predict support ticket escalations in a large software organization. Conducted in collaboration with IBM, the study follows a Design Science methodology that iteratively builds and evaluates artifacts with industry stakeholders. An ethnographic field study at IBM’s Victoria site, supplemented by semi‑structured interviews with analysts and managers worldwide, revealed that escalation risk assessment relies on a blend of customer‑centric, issue‑centric, and interaction‑centric cues: business importance of the client, historical dissatisfaction patterns, emotional tone, ticket severity, priority, response latency, hand‑offs, and analyst workload, among others.

These insights guided a systematic feature‑engineering effort that produced a “Support Ticket Model” comprising roughly twenty quantitative features grouped into four categories: (1) Customer Profile (size, contract type, prior escalations), (2) Issue Characteristics (severity, type, reproducibility), (3) Interaction History (response times, communication frequency, sentiment scores, number of escalations), and (4) Support Team Capacity (analyst experience, current load, average resolution time). All features are derivable from IBM’s internal ticket repository and associated logs, enabling automated extraction.

Using a dataset of over 2.5 million tickets and 10,000 escalation instances, the authors trained an ensemble of Gradient Boosting and Random Forest classifiers, employing cost‑sensitive learning and sampling techniques to address the severe class imbalance (escalations < 0.5 %). The resulting model achieved a recall of 79.9 %, precision of 45 %, and an F1‑score of 57 %. Crucially, by focusing analyst attention on the top 10 % of tickets flagged as high‑risk, the system could pre‑emptively capture more than 80 % of eventual escalations, translating into an 80.8 % reduction in manual workload for escalation monitoring.

A prototype web‑based dashboard was deployed in IBM’s weekly ticket‑triage meetings. Field evaluation showed that meetings became 30 % shorter, decision‑making was more transparent, and a modest increase (≈5 pp) in customer‑satisfaction survey scores was observed. The authors argue that the feature set is portable to other large enterprises with comparable support processes, and they outline future extensions such as incorporating natural‑language sentiment analysis, time‑series forecasting, and multimodal data (chat, voice) to further boost predictive performance.

Overall, the study demonstrates that embedding analysts’ domain expertise into a data‑driven model yields a practical, high‑recall escalation‑prediction tool that reduces operational costs, improves support efficiency, and helps maintain customer happiness in large‑scale software organizations.