Standardizing reference data in gap statistic for selection optimal number of cluster in K-means algorithm

Abstract

Clustering aims to partition data into distinct groups where the data points within each cluster are highly similar, while remaining dissimilar to those in other clusters. The K-means algorithm is widely recognized for its effectiveness in clustering; however, a significant limitation is its inability to automatically determine the optimal number of clusters (ONC). To address this issue, various methods have been developed, including the Gap Statistic (GS), which helps identify the ONC. Despite its utility, the GS method can yield inconsistent results due to variations in the reference data units, leading to skewed comparisons and potential misestimation of the ONC. This paper introduces the Enhanced Gap Statistic (EGS), which standardizes the reference data within the GS framework, improving both the accuracy and efficiency of ONC determination. The experimental results show that the EGS method outperforms traditional techniques such as Fuzzy Clustering, Silhouette, Elbow, and the original Gap Statistic, with accuracy and execution times of 79.58 % and 0.21 seconds (Hitter’s dataset), 88.58 % and 0.01 seconds (Time Series), 91.58 % and 1.24 seconds (Well Log), and 96.58 % and 2.14 seconds (Traffic Crash dataset). The hypothesis testing of EGS shows high significance, with p-values less than 0.05 across different datasets, and the coefficient of determination (R²) is significantly higher compared to the Gap Statistic. These results demonstrate that EGS consistently delivers superior performance, particularly for larger datasets, offering both more accurate ONC determinations and faster execution times compared to existing methods. By eliminating biases associated with varying measurement units, the EGS method ensures more reliable and consistent clustering outcomes compared to the traditional GS approach.

Topics & Concepts