A Multi-Criteria Automated MLOps Pipeline for Cost-Effective Cloud-Based Classifier Retraining in Response to Data Distribution Shifts

The performance of machine learning (ML) models often deteriorates when the underlying data distribution changes over time, a phenomenon known as data distribution drift. When this happens, ML models need to be retrained and redeployed. ML Operations (MLOps) is often manual, i.e., humans trigger the process of model retraining and redeployment. In this work, we present an automated MLOps pipeline designed to address neural network classifier retraining in response to significant data distribution changes. Our MLOps pipeline employs multi-criteria statistical techniques to detect distribution shifts and triggers model updates only when necessary, ensuring computational efficiency and resource optimization. We demonstrate the effectiveness of our framework through experiments on several benchmark anomaly detection data sets, showing significant improvements in model accuracy and robustness compared to traditional retraining strategies. Our work provides a foundation for deploying more reliable and adaptive ML systems in dynamic real-world settings, where data distribution changes are common.

💡 Research Summary

This paper presents the design, implementation, and evaluation of an automated MLOps pipeline for cost-effective retraining of neural network classifiers in response to data distribution drift. The core challenge addressed is the performance degradation of machine learning models deployed in production when the statistical properties of incoming data evolve over time, a phenomenon known as data drift.

The authors propose a pipeline that moves beyond manual or periodic retraining by implementing an intelligent, condition-based trigger. This trigger relies on a multi-criteria statistical framework to detect significant distribution shifts. It continuously monitors incoming data against a reference (training) distribution using a battery of metrics: the Kolmogorov-Smirnov (KS) test statistic, Kullback-Leibler (KL) Divergence, Population Stability Index (PSI), Maximum Mean Discrepancy (MMD), and changes in model performance metrics (Accuracy and F1-score). These individual metrics are combined into a single, comprehensive drift score (D_S) via a weighted linear combination. Retraining is initiated only when this aggregated score exceeds a predefined threshold (τ), ensuring computational resources are used only when necessary and avoiding the costs associated with unnecessary retraining cycles.

The pipeline architecture fully embraces CI/CD principles. It consists of key components: a production data stream, reference training data, a data drift detector, a data mixer, and automated CI/CD stages. Upon detection of significant drift, the data mixer creates an updated training dataset by blending new data with the historical data, which then feeds into an automated process of model retraining, validation, packaging (e.g., into a Docker image), testing in staging environments, and final deployment to production.

The experimental investigation validates the pipeline’s effectiveness using seven benchmark anomaly detection datasets (e.g., CICIDS, Credit Card Fraud) to simulate various data drift scenarios. An autoencoder serves as the base classifier. The proposed method, dubbed “Auto-MLOps,” is compared against three baselines: a static model with no retraining (STATIC), a model retrained at fixed intervals (FIXED), and a model retrained naively upon any drift detection without cost optimization (NAIVE).

Results demonstrate that Auto-MLOps maintains higher and more stable model accuracy as the severity of data drift increases, compared to the baseline strategies. Crucially, it achieves this robust performance with a lower retraining frequency than the FIXED and NAIVE approaches. This directly translates to reduced cloud computational costs, as retraining is both less frequent and more strategically timed. The work highlights two main contributions: 1) an enhanced, systematic model monitoring system that diagnostically pinpoints when and why performance degrades using rigorous statistical evidence, and 2) a scientifically robust methodology for automating the model lifecycle, providing a verifiable and adaptable standard for cost-aware, drift-responsive MLOps.