Visibility Convolutional Neural Networks for Label Distribution Learning of Facial Emotions

Human emotion recognition from facial expressions depicted in images is an active area of research in computer vision and machine learning. Facial expressions usually represent a mix of basic emotions, each with a different intensity. Hence, measuring these intensities from facial expression images is a challenging task. Previous studies have solved this task by using label-distribution learning (LDL). Models based on LDL predict a probability distribution for each face image, where the distribution represents the intensities of the emotions depicted in the image. In this paper, we propose an LDL framework that introduces a novel convolutional (Conv) layer called the visibility convolutional layer (VCL). The VCL preserves the advantages of traditional Conv layers, in terms of using filters to extract features, while reducing the number of learnable parameters and allowing for the extraction of strong texture features. Our LDL framework, which we call VCNN-ELDL, uses the features extracted by the VCLs and those extracted by traditional Conv layers to predict a probability distribution for each image using supervised learning. We evaluate VCNN-ELDL on the widely-used$s$-JAFFE and$s$-BU-3DFE datasets, which are annotated with probability distributions as the ground truth representing the mix of six basic human emotions, i.e.,happiness, sadness, anger, fear, surprise,anddisgust. Our results show that our framework can accurately predict the probability distribution representing the mix of emotions depicted in face images by attaining a stronger performance than state-of-the-art LDL methods.