NannyML is an open-source Python library designed to monitor and maintain the performance of machine learning models in production environments, even when ground truth labels are delayed or unavailable. By estimating model performance without access to target data, NannyML enables data scientists and ML engineers to detect and address issues such as data drift and concept drift, ensuring that models continue to deliver accurate and reliable predictions over time. Key Features and Functionality: - Performance Estimation Without Labels: Utilizes advanced statistical techniques like Confidence-Based Performance Estimation (CBPE) and Direct Loss Estimation (DLE) to estimate classification and regression metrics in real-time, without requiring actual outcomes. - Data Drift Detection: Monitors changes in input data distributions at both feature and dataset levels, employing metrics such as Jensen-Shannon divergence and Wasserstein distance to identify shifts that may impact model performance. - Concept Drift Detection: Measures the impact of concept drift on model performance, providing insights into when retraining may be necessary. - Root Cause Analysis: Links data drift alerts to specific features, enabling rapid identification of the factors contributing to performance degradation. - Business Impact Assessment: Allows users to define cost-benefit matrices to quantify the monetary or business-oriented outcomes of model performance, ensuring alignment with organizational objectives. - Automated Monitoring and Alerting: Provides intelligent alerting systems that focus on performance-impacting issues, reducing false alarms and alert fatigue. - Integration and Deployment: Offers seamless integration with existing MLOps pipelines and can be deployed within an organization's cloud infrastructure for enhanced security and control. Primary Value and Problem Solved: NannyML addresses the critical challenge of maintaining machine learning model performance in production settings where ground truth labels are often delayed or absent. By enabling real-time performance estimation and drift detection without the need for actual outcomes, NannyML empowers data science teams to proactively identify and resolve issues that could lead to model degradation. This proactive monitoring ensures that models continue to provide accurate predictions, thereby preserving their business value and supporting informed decision-making processes. Additionally, by reducing false alarms and focusing on performance-impacting changes, NannyML helps teams avoid alert fatigue and allocate resources more effectively.