Differential entropy and correlation analysis: a system approach

Every year we see an increase in the requirements for the efficiency of both newly created and existing systems. This leads to a complication of the tasks to be solved when creating, diagnosing, and managing these systems and phenomena. One of these tasks is to determine the nature and structure of relationships between ongoing processes. These processes are stochastic in nature, and they are usually described by correlation dependencies. However, their multidimensionality and multiple connectivity make it difficult to apply correlation analysis. For a clearer understanding and use of correlation dependencies, their systematic interpretation is necessary. A fundamental characteristic of any system with ambiguous or probabilistic behavior is entropy. Specifically, several results have been obtained linking the differential entropy of random vectors with correlation characteristics. The purpose of the article is to investigate and systematize the relationship between differential entropy and correlation indicators used in multidimensional statistical analysis. The article considers all the main variants of correlations in multidimensional systems, including the cross correlation between all elements, between subsystems, at different levels of the system, between one element (subsystem) and a group of elements (subsystems). In all cases, it has been established that correlation indices are analytically determined from the differential entropy of a system considered in the form of a random vector. Several examples have been given to illustrate the results obtained.