Beyond classification accuracy: Uncertainty-aware deep learning for coral reef monitoring

Deep learning has rapidly gained popularity for automated image classification tasks, and is now becoming a default standard for use in automated ecosystem monitoring applications. Although there have been many published articles on the use of deep learning classifiers for image classification in ecology, including for coral reefs, there has been little published in the literature on quantifying classification uncertainty and error propagation including how this influences time series monitoring and coral cover assessment. We investigate a method for minimizing error from deep learning predictions, with a specific application to prediction of coral reef communities from benthic survey images. We show this through both simulated data distributions, and real world datasets, comparing recommended best practice to our proposed method of unbalanced training of calibrated classifiers with Monte Carlo (MC) sampling. Our results show that the composition of the underlying training data distribution can have major ramifications for the application of deep learning classifiers in the wild. We also show the use of MC sampling of a calibrated model's prediction probabilities is a more accurate method for estimating cover percentage, and propagating uncertainty from a model, especially under noisy conditions we find in ecology. Unbalanced training coupled with MC sampling provides a more accurate assessment of coral cover prediction from deep learning classifiers, especially for classes which have high predictive uncertainty. The results of this paper provide a method for practitioners of automation at large scale to reduce error in automated abundance prediction, and be more confident in the use of deep learning for change detection in coral reef monitoring, and ecological studies more broadly.