Stylized Explanations: Enhancing Neural Networks Visual Interpretations Using Neural Style Transfer

Neural networks have made significant achievements in areas like computer vision and natural language processing. However, there is an increasing demand to understand their outputs and decisions. They are considered black box models because it is hard to understand their complex architecture as they lack justification and transparency. In this paper, we present a novel approach by generating stylized explanations using neural style transfer. Unlike methods that use one input image to generate activation maps, we adopt the neural style transfer technique to generate artistic styles of the input image. We select eight styles from the STaDA experiments, “la_muse”, “rain_princess”, “sunflower”, “the_scream”, “the_shipwreck”, “udnie”, “wave”, and “your_name”. We pass the stylized images with the input image to Score-CAM and produce class activation maps for the nine images, the input image and the eight stylized images. We evaluate the stylized activation maps by conducting experiments like faithfulness, object localization, and sanity check. The input image activation map outperformed stylized activation maps in terms of faithfulness with a lower confidence drop. However, in terms of object localization, “sunflower”, “the_scream”, “the_shipwreck”, “udnie”, “wave”, and “your_name” stylized activation maps outperformed the input image activation map with a higher IoU value. In terms of sanity check, both the input image activation map and stylized activation maps were sensitive to VGG-16 model randomization.