Alternate Loss Functions for Classification and Robust Regression Can Improve the Accuracy of Artificial Neural Networks

All machine learning algorithms use a loss, cost, utility, or reward function to encode the learning objective and oversee the learning process. This paper shows that the training speed, final accuracy and robustness to outliers can significantly depend on the loss function used to train neural networks. In particular derivative of the loss with respect to the output can be significantly different with different loss functions leading to significantly different performance after gradient-descent based training. This paper explores the possible advantages of using new loss functions that penalize errors differently compared to the popular Cross-entropy and Mean squared loss on 7 benchmarks. Two new loss functions for classification are proposed and switching loss functions during training is shown to significantly speedup training. Two new infinitely differentiable loss function that more closely approximate the absolute error loss compared to the Huber and Log-Cosh losses used for robust regression are proposed and shown to significantly outperform Squared error, Huber, and Log-Cosh losses on datasets contaminated with outliers.