An Optimized Hybrid K-Means Algorithm for High- Dimensional Data Clustering

Abstract:The proliferation of high-dimensional data across computational domains has rendered traditional partitioning-based clustering algorithms increasingly ineffective due to the "curse of dimensionality." This paper proposes anovel optimized hybrid K-means framework designed to overcome the limitations of distance metric degradationand local optima entrapment. Our methodology integrates Deep Representation Learning using VariationalAutoencoders (VAEs) to map sparse high-dimensional features into a regularized latent manifold, followed by aglobal-local search optimization leveraging Particle Swarm Optimization (PSO) and K-means++ initialization. Tofurther enhance cluster cohesion, a hybrid distance strategy combining weighted Cosine and Manhattan metricsis employed, alongside a Z-score-based post-clustering refinement mechanism. Experimental results onbenchmark datasets, including the Breast Cancer Wisconsin (BCW) and CIC IoT-DIAD 2024 datasets,demonstrate significant performance gains. The proposed algorithm achieved an accuracy of 98.25% on medicaldata and 98.99% on cybersecurity IoT traffic, with a high Silhouette Coefficient of 0.752. This hybrid approachprovides a robust, scalable, and highly accurate solution for complex data landscapes where traditional Euclidean-based methods fail.