Absolute indices for determining compactness, separability and number of clusters

Identifying “true” clusters in a dataset is inherently challenging. Clustering models and algorithms may fail to produce compact, well-separated groups or to correctly determine the optimal number of clusters. Cluster validity indices are often applied to find such clusters. However, most existing indices are relative measures, designed primarily for comparing clustering algorithms or tuning their parameters. In this paper, we introduce new absolute cluster validity indices that assess both cluster compactness and separability. For each cluster, we define a compactness function, and for each pair of clusters, we define an associated set of neighboring points. The compactness function measures the cohesion of individual clusters as well as the entire clustering distribution, while the neighboring-point sets are used to define the margin between cluster pairs and the overall distribution margin. These compactness and separability indices are then employed to estimate the true number of clusters. We evaluate the proposed indices on a variety of synthetic and real-world datasets and compare their performance with several widely used cluster validity measures.