Smart data-driven medical decisions through collective and individual anomaly detection in healthcare time series

Abstract

• Developed a novel approach for detecting anomalies in healthcare data using unsupervised machine learning: The study introduces a dual-strategy approach for anomaly detection in healthcare time series data. • Identified collective anomalies across multiple healthcare practice centres: The research detects anomalies where multiple practice centres collectively exhibit unusual patterns, indicating systemic irregularities that require further investigation. • Focused on enhancing healthcare quality and decision-making: By identifying anomalies early, the method enables timely interventions, enhances healthcare quality, and supports efficient decision-making within the healthcare sector. • Maintained privacy and confidentiality of healthcare data: The approach ensures privacy by not requiring personal information, while providing accessible visual representations of anomalies, thereby supporting non-technical users in understanding healthcare data insights. Anomalies in healthcare refer to deviation from the norm of unusual or unexpected patterns or activities related to patients, diseases or medical centres. Detecting these anomalies is crucial for timely interventions and efficient decision-making, helping to identify issues like operational inefficiencies, fraud and emerging health complications. This study presents a novel method for detecting both collective and individual anomalies in healthcare data through time series analysis using unsupervised machine learning. The dual-strategy approach leverages two methodologies: a ’practice centre-based approach’ which monitors changes across different practice centres and a ’process-based approach’ which focuses on identifying anomalies within individual centres. The former allows for early detection of systemic issues, while the latter highlights specific irregularities within a centre’s operations. The study utilised a dataset over 500,000 medical records from multiple GP practice centres in the UK collected between 2018–2023. Data are clustered using DBSCAN to identify collective anomalies from deviations from linear trends in consecutive two-month scatterplots. Individual anomalies are identified by examining the SOM-clustered time series of various medical processes within a specific practice centre, where graphs show deviation from the typical pattern. Our approach addresses some challenges posed by the complexity and sensitivity of healthcare data by not requiring personal information. The method offers accurate visual representations making the data accessible and interpretable for non-technical users. Unlike traditional methods focusing solely on subsequence anomalies, our technique analyses the collective behaviour across multiple time series providing a more comprehensive perspective. This study underscores the importance of integrating unsupervised anomaly detection with clinical expertise to ensure that statistically anomalous patterns align with clinical relevance. The dual-strategy clustering method holds significant potential for enabling timely interventions, proactively identifying potential crises, and ultimately contributing to better decision-making and operational efficiency within the healthcare sector.

Topics & Concepts