Information Theory

Information Theory is as an important foundation for data science, providing the logical validation for how information is quantified, stored, and distributed. Information theory is an application of probability to information (data). The fundamental components of information are called symbols . In some encoding schemes, symbols may be analogous to atomic or non-divisible data such as 0 or 1 bits, numbers, or characters. In other schemes, symbols may represent abstract words, expressions, or labels for a combination of machine states. Information may represent a finite state machine, a model, a data structure, a protocol, a language, or a transmission of any of these. A series of symbols will convey information, a signal or message, and non-information, noise, in varying proportions. The greater the amount of information relative to noise, the greater the likelihood that the information is decipherable.