Some comments on the relationship between the rate function in the large deviation principle and the Kullback–Leibler divergence: toward the interpretation of neural estimation of mutual information
Abstract
The recent explosive development of artificial intelligence is largely driven by deep neural networks such as Transformers. Computational capabilities of deep neural networks have advanced rapidly, and their applications have expanded across a wide range of scientific fields. Motivated by the precise computation of mutual information by deep neural networks, named mutual information neural estimation (MINE), we investigate the relationship between the rate function in the large deviation principle and the Kullback–Leibler divergence, and prove an equality between them. We further cast an interpretation of MINE in terms of the rate function.
Fields of Study & Subjects
Statistical Mechanics and Entropy
Adversarial Robustness in Machine Learning
Explainable Artificial Intelligence (XAI)
Publication details
- Journal / source
- Japan Journal of Industrial and Applied Mathematics
- Publisher
- Springer Science and Business Media LLC
- Volume
- 43
- Issue
- 1
- Publication date
- 31 January 2026
- Language
- EN