Detecting anomalies in log files using the Damerau-Levenshtein distance metric

Abstract In recent decades, significant research effort has been put into developing solutions to support automatized or semi-automatized analysis of log files. A large number of algorithms appeared based on neural networks. In this paper we introduce a new approach to anomaly detection in log files, that does not rely on neural networks. The building blocks of our approach have been well-known in machine learning for a long time. We propose to use a weighted Damerau-Levenshtein distance metric to quantify the similarity between log sequences. We introduce a kNN-based algorithm for semi-supervised log anomaly detection, and an HDBSCAN-based solution for the unsupervised problem. For the latter, we extend the algorithm by incorporating a manual feedback mechanism, enabling human domain experts to modify sequence labels when necessary.