Differential Shannon and Rényi entropies revisited
Abstract
Abstract Shannon entropy for discrete distributions is a fundamental and widely used concept, but its continuous analogue, known as differential entropy, lacks essential properties such as positivity and compatibility with the discrete case. In this paper, we analyze this incompatibility in detail and illustrate it through examples. To overcome these limitations, we proposemodified versions of Shannon and Rényi entropy that retain key properties, including positivity, while remaining close to the classical forms. We also define compatible discrete functionals and study the behavior of the proposed entropies for the normal and exponential distributions.
Fields of Study & Subjects
Statistical Mechanics and Entropy
Mathematical Inequalities and Applications
Probability and Statistical Research
Publication details
- Journal / source
- Lithuanian Mathematical Journal
- Volume
- 66
- Issue
- 1
- Pages
- 128–150
- Publication date
- 1 January 2026
- Language
- EN