LogBoost: Boost Log Anomaly Detection by Cherry-Picking Log Sequences

Debugging and operating services always benefit from logs. Since logs provide rich information on events and render comprehensive execution traces, it is imperative to automatically detect faults from extensive logs through log-based anomaly detection. However, due to the ineffectiveness and complexity of log-based detection models, they have not been widely adopted in template evaluation and real-world detection. Therefore, we propose LogBoost, a lightweight framework to boost log-based anomaly detection by automatically reducing redundant log templates. Based on our proposed similarity measurement, it effectively sorts the importance of log templates and identifies templates that are ineffective in anomaly detection. In evaluation, we introduce Spark-SDA, a new dataset featuring more diverse log templates and excessively long sequences, alongside the HDFS log dataset. We further evaluate LogBoost using established log-based anomaly detection models. The results demonstrate that eliminating ineffective log templates via LogBoost improves feature efficacy while reducing computational overhead. For instance, RandomForest obtains an F1-score at 0.983 given only 500 training samples on an optimized frequency vector of the HDFS dataset. Meanwhile, the lengths of log sequences are reduced by 51% to 77%, and the prediction time of deep learning models is reduced by 55% to 81%. Our results show that LogBoost is an effective approach to accelerate log-based anomaly detection.