Improving Anomaly Detection in the HDFS Dataset with Novel Machine Learning Models and Techniques

With the growing scale and complexity of log data, manual anomaly detection has become increasingly time-consuming and error-prone, necessitating the development of robust machine learning-based solutions. The HDFS (Hadoop Distributed File System) dataset, a large-scale real-world log collection, serves as a standard benchmark for evaluating both supervised and unsupervised anomaly detection methods. Available in two variants - a reduced and a complete version - this dataset facilitates comprehensive performance comparisons. Our empirical analysis reveals significant limitations in existing approaches: supervised methods exhibit poor performance on the reduced dataset, while most unsupervised techniques underperform across both versions. To address these shortcomings, we introduce several novel machine learning approaches for log-based anomaly detection. Additionally, we investigate the effects of alternative feature extraction techniques. We also examine the application of Synthetic Minority Over-sampling Technique (SMOTE) to mitigate class imbalance in supervised learning, as well as the incorporation of temporal features encoding inter-log time intervals. Our experimental results demonstrate that the proposed methods achieve statistically significant improvements in detection accuracy over existing approaches on both HDFS dataset variants, establishing new benchmarks for log-based anomaly detection.