Performance of test cases for machine learning classifier: coverage perspective

Abstract A rapid rise in machine learning-based applications has made it one of the most popular areas in the field of artificial intelligence (AI). The most commonly used libraries to implement the algorithms used in these applications are Scikit learn and Weka. It is challenging to test these machines learning based applications due to the Oracle Problem. The problem is when the expected outcome is not known and hence the testing of such applications cannot be performed via traditional testing techniques. One of the solution to the Oracle problem is the use of Metamorphic testing to test the machine learning applications. The code of machine learning algorithms is often ignored, when testing of ML-based applications is done. However, the usage of the machine learning algorithms within the libraries requires formal testing to improve reliability. This work evaluates the Metamorphic relations for machine learning algorithms by finding their kill rate while testing 5 machine learning (ANN, ID3, KNN, Naive Bayes, SVM) classifiers from the Scikit Learn library. This work also calculates the statement coverage, while testing the metamorphic relations. The relationship between the effectiveness of fault detection and code coverage is identified as well.