Enhancing concept-based image classification via knowledge reasoning

Deep learning has made great progress in supervised image classification. However, a persistent challenge in this field is the need for better explainability. This is crucial for building trust, troubleshooting, and ensuring regulatory compliance, ultimately leading to more responsible and effective AI applications. To tackle this challenge, we have developed a deep learning framework based on concepts, a recent method aimed at enhancing explainability, albeit with less emphasis on image classification performance. Our innovation lies in integrating a specific knowledge graph as a reasoning tool. This approach leverages the interdependence between concepts and classes to improve the performance of the concept-based model, achieving a balance between explainability and efficiency. Results from both synthetic and real data validate the effectiveness of our approach in balancing efficiency and explainability. Additionally, our model supports human test-time intervention to update its final prediction after incorporating new expert feedback. Our experiments show significant improvements in both classification and concept efficiencies.