Locality Guided Neural Networks for Explainable Artificial Intelligence

In current deep learning architectures, each of the deeper layers in networks tends to contain hundreds of unorganized neurons, which makes it hard for humans to understand how they interact with each other. Research in Explainable Artificial Intelligence (XAI) aims to alleviate the black-box nature of current AI methods and make them understandable by humans. In this dissertation, we propose a novel algorithm called Locality Guided Neural Network (LGNN) for XAI. Motivated by Self-Organizing Maps (SOMs), the goal is to enforce a local topology on a layer of a deep network such that neighboring neurons are organized using correlation as a criterion. Our algorithm does not change the structure of a model, so it can be easily plugged into current state-of-the-art Convolutional Neural Network (CNN) models. A cluster of neighboring neurons activating for a class makes the network both quantitatively and qualitatively more interpretable when visualized, as we show through experiments. Additionally, user studies demonstrate how LGNN visualizations can allow users to intuit a network’s understanding of each class within a dataset. In the experiments, we train VGG and WRN networks for image classification on CIFAR100 and Imagenette datasets. Our experimental results show that LGNN maintains competitive classification accuracy to baseline models within 0.5%. Additionally, case studies and user experiments explore how LGNN can be utilized for downstream tasks such as manually selecting neurons for feature visualization. Our user study tasked users with predicting test set classification accuracy using visualizations from both LGNN and Uniform Manifold Approximation and Projection (UMAP), which is state-of-the-art. The users had a mean correlation of 0.6285 for LGNN and 0.5913 for UMAP between the user predictions and the actual test set accuracy. These correlations demonstrate that LGNN visualizations can be utilized alongside state-of-the-art visualization toolsto analyze neural networks.