On the entropy of some classes of distributions and their mixtures -

Abstract

Entropy is an important tool mainly in information theory, but also in machine learning, biomedical engineering and other fields. Specifically in information theory, it is used in the analysis of a channel's capacity, data compression and the rate distortion function of information sources, among other applications. Expressions for Shannon's entropy are commonly known for discrete or continuous random variables. Through this work, we formulate a candidate expression which can be applied to additional types of random variables and their mixtures with a special emphasis on mixtures of discrete and continuous random variables, commonly encountered in real world signal processing applications. With this new definition of entropy, we also introduce a corresponding order relation. The order relation helps in comparing different entropies of different mixture random variables. In addition, the new definition of entropy is then applied to the rate distortion problem of mixture sources. The defined order relation is used in searching for the rate distortion function of a mixture source. Finally, the definition of entropy is used in some common and frequent applications such as: the maximization of entropy subject to certain constraints, and the asymptotic equipartition property.