Quantifying Uncertainty in Machine Learning-Based Pervasive Systems: Application to Human Activity Recognition

Quantifying Uncertainty in Machine Learning-Based Pervasive Systems: Application to Human Activity Recognition Vladimir BALDITSYN LCIS, Grenoble INP - UGA, Valence, France vladimir.balditsyn@imag.fr Philippe LALANDA Univ. Grenoble Alpes, Grenoble, France philippe.lalanda@imag.fr German VEGA Univ. Grenoble Alpes, Grenoble, France german.vega@imag.fr Stéphanie CHOLLET LCIS, Grenoble INP - UGA, Valence, France stephanie.chollet@esisar.grenoble-inp.fr Abstract—The recent convergence of pervasive computing and machine learning has given rise to numerous services, impacting almost all areas of economic and social activity. However, the use of AI techniques precludes certain standard software development practices, which emphasize rigorous testing to ensure the elimination of all bugs and adherence to well- defined specifications. ML models are trained on numerous high- dimensional examples rather than being manually coded. Conse- quently, the boundaries of their operating range are uncertain, and they cannot guarantee absolute error-free performance. In this paper, we propose to quantify uncertainty in ML-based systems. To achieve this, we propose to adapt and jointly utilize a set of selected techniques to evaluate the relevance of model predictions at runtime. We apply and evaluate these proposals in the highly heterogeneous and evolving domain of Human Activity Recognition (HAR). The results presented demonstrate the relevance of the approach, and we discuss in detail the assistance provided to domain experts. Index Terms—Machine Learning, Uncertainty quantification, Human Activity Recognition. I. INTRODUCTION The recent convergence of pervasive computing and ma- chine learning (ML) has led to the creation of numerous highly valuable and sophisticated services. This development is transforming almost every sector of economic and social activity [1]. For instance, it is particularly evident in the fields of industry, with the advent of Industry 4.0, healthcare, trans- portation, or smart residential environments. This integration is, therefore, making a significant impact on both businesses and everyday life. However, the reliance on ML-based services introduces sev- eral fundamental challenges. In particular, it implies a major conceptual shift from traditional development methodologies, which emphasize rigorous testing as a crucial step to ensure the elimination of all bugs and adherence to well-defined specifications. ML models, on the other hand, are trained on examples (usually numerous and of high dimension) rather than being manually coded. As a result, the boundaries of their operating range are uncertain, making it impossible to formally prove that specific constraints are consistently met. This raises concerns about reliability, safety, and account- ability, as ML-based solutions may behave unpredictably un- der certain conditions. They are particularly sensitive to noisy or inaccurate data, and more generally to differences between the data distribution at training time and at inference time. The different types of distributional shifts are generally categorized as follows: • Sample selection bias that occurs when the training data is not a representative sample of the population, leading to skewed model predictions. • Covariate shift that occurs when the input feature distri- bution changes between training and deployment while their relationship with the target remains constant. • Label shift that occurs when the distribution of the labels changes between training and deployment, while the class-conditional distributions remain constant. • Domain shift that occurs when models are applied to a different domain than they were trained on, involving changes in both input features and target distributions. This sensitivity can lead to significant performance degrada- tion, as the model may not generalize well to new, unseen data. Furthermore, small deviations in the data can accumulate and result in substantial errors, making it challenging to maintain the desired quality of service. It is therefore important to provide a level of uncertainty with every prediction, espe- cially in practical fields like healthcare or industrial automa- tion, where wrong decisions can have significant negative consequences. Understanding the level of uncertainty helps users assess the reliability of a prediction and manage risks effectively. When uncertainty is high, decision-makers can proceed with caution or seek additional information before making a critical decision. Conversely, when uncertainty is low, decisions can be made more confidently. For instance, in predictive maintenance within Industry 4.0 [2], if a model predicts a malfunction with low uncertainty, operators can con- fidently plan maintenance. However, high uncertainty prompts operators to conduct additional tests or gather further evidence before committing to costly operations. To address this issue, the field of uncertainty quantification (UQ) has see

This content is AI-processed based on open access ArXiv data.