There are close parallels between the mathematical expressions for the thermodynamic entropy, usually denoted by S, of a physical system in the statistical thermodynamics established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s; and the information-theoretic entropy, usually expressed as H, of Claude Shannon and Ralph Hartley developed in the 1940s. In Thermodynamics (a branch of Physics) entropy, symbolized by S, is a measure of the unavailability of a system ’s Energy In Thermodynamics, statistical thermodynamics is the study of the microscopic behaviors of Thermodynamic systems using Probability theory. Ludwig Eduard Boltzmann ( February 20, 1844 &ndash September 5, 1906) was an Austrian Physicist famous for his founding Josiah Willard Gibbs ( February 11, 1839 &ndash April 28, 1903) was an American theoretical Physicist, Chemist Events and Trends Technology The invention of the prototype telephone by Alexander G Claude Elwood Shannon (April 30 1916 – February 24 2001 an American Electronic engineer and Mathematician, is "the father of Information Ralph Vinton Lyon Hartley ( November 30, 1888 – May 1, 1970) was an Electronics researcher The 1940s decade ran from 1940 to 1949 Events and trends The 1940s was a period between the radical 1930s and the conservative 1950s which also leads the period to be Shannon, although not initially aware of this similarity, commented on it upon publicizing information theory in A Mathematical Theory of Communication. "A Mathematical Theory of Communication" is an influential 1948 article by Mathematician Claude E

This article explores what links there are between the two concepts, and how far they can be regarded as connected.

Equivalence of form of the defining expressions

Discrete case

Boltzmann's tombstone, featuring his equation S = k log W

The defining expression for entropy in the theory of statistical mechanics established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s, is of the form:

where p_i is the probability of the microstate i taken from an equilibrium ensemble. In Thermodynamics (a branch of Physics) entropy, symbolized by S, is a measure of the unavailability of a system ’s Energy Statistical mechanics is the application of Probability theory, which includes mathematical tools for dealing with large populations to the field of Mechanics Ludwig Eduard Boltzmann ( February 20, 1844 &ndash September 5, 1906) was an Austrian Physicist famous for his founding Josiah Willard Gibbs ( February 11, 1839 &ndash April 28, 1903) was an American theoretical Physicist, Chemist Events and Trends Technology The invention of the prototype telephone by Alexander G

The defining expression for entropy in the theory of information established by Claude E. Shannon in 1948 is of the form:

where p_i is the probability of the message m_i taken from the message space M. Information theory is a branch of Applied mathematics and Electrical engineering involving the quantification of Information. Claude Elwood Shannon (April 30 1916 – February 24 2001 an American Electronic engineer and Mathematician, is "the father of Information Year 1948 ( MCMXLVIII) was a Leap year starting on Thursday (link will display the 1948 calendar of the Gregorian calendar.

Mathematically H may also be seen as an average information, taken over the message space, because when a certain message occurs with probability p_i, the information -log(p_i) will be obtained.

If all the microstates are equiprobable (a microcanonical ensemble), the statistical thermodynamic entropy reduces to the form on Boltzmann's tombstone,

where W is the number of microstates. The microcanonical ensemble is the simplest of the ensembles of Statistical mechanics.

If all the messages are equiprobable, the information entropy reduces to the Hartley entropy

where | M | is the cardinality of the message space M. The Hartley function is a measure of uncertainty introduced by Ralph Hartley in 1928 In Mathematics, the cardinality of a set is a measure of the "number of elements of the set"

The logarithm in the thermodynamic definition is the natural logarithm. The natural logarithm, formerly known as the Hyperbolic logarithm is the Logarithm to the base e, where e is an irrational It can be shown that the Gibbs entropy formula, with the natural logarithm, reproduces all of the properties of the macroscopic classical thermodynamics of Clausius. Classical thermodynamics is a branch of Physics developed in the nineteenth century by Sadi Carnot (1824 Emile Clapeyron (1834 Rudolf Clausius Rudolf Julius Emanuel Clausius (Born Rudolf Gottlieb, January 2, 1822 &ndash August 24, 1888) was a German Physicist (See article: Entropy (statistical views)). In Thermodynamics, statistical entropy is the modeling of the energetic function Entropy using Probability theory.

The logarithm can also be taken to the natural base in the case of information entropy. In Mathematics, the logarithm of a number to a given base is the power or Exponent to which the base must be raised in order to produce This is equivalent to choosing to measure information in nats instead of the usual bits. A nat (sometimes also nit or even nepit) is a Logarithmic unit of Information or entropy, based on Natural logarithms and In practice, information entropy is almost always calculated using base 2 logarithms, but this distinction amounts to nothing other than a change in units. One nat is about 1. 44 bits.

The presence of Boltzmann's constant k in the thermodynamic definitions is a historical accident, reflecting the conventional units of temperature. Bridge from macroscopic to microscopic physics Boltzmann's constant k is a bridge between Macroscopic and microscopic physics It is there to make sure that the statistical definition of thermodynamic entropy matches the classical entropy of Clausius, thermodynamically conjugate to temperature. Temperature is a physical property of a system that underlies the common notions of hot and cold something that is hotter generally has the greater temperature For a simple compressible system that can only perform volume work, the first law of thermodynamics becomes

But one can equally well write this equation in terms of what physicists and chemists sometimes call the 'reduced' or dimensionless entropy, σ = S/k, so that

Just as S is conjugate to T, so σ is conjugate to kT (the energy that is characteristic of T on a molecular scale). In Thermodynamics, the first law of thermodynamics is an expression of the more universal physical law of the Conservation of energy.

Continuous case

The most obvious extension of the Shannon entropy is the differential entropy,

As long as f(x) is a probability density function, p. Differential entropy (also referred to as continuous entropy) is a concept in Information theory which tries to extend the idea of (Shannon entropy d. f. , H repesents the average information (entropy, disorder, diversity etcetera) of f(x). In Thermodynamics (a branch of Physics) entropy, symbolized by S, is a measure of the unavailability of a system ’s Energy For any uniform p. d. f. f(x), the exponential of H is the volume covered by f(x) (in analogy to the cardinality in the discrete case). In Mathematics, the cardinality of a set is a measure of the "number of elements of the set" The volume covered by a n-dimensional multivariate Gaussian distribution with moment matrix M is proportional to the volume of the ellipsoid of concentration and is equal to square root{ (2 pi e)ⁿ determinant(M) }. MVN redirects here For the airport with that IATA code in Mount Vernon Illinois, see Mount Vernon Airport. In Mathematics, a square root of a number x is a number r such that r 2 = x, or in words a number r whose In Algebra, a determinant is a function depending on n that associates a scalar, det( A) to every n × n The volume is always positive.

Average information may be maximized using Gaussian adaptation - one of the evolutionary algorithms - keeping the mean fitness - i. Gaussian adaptation (GA is an Evolutionary algorithm designed for the maximization of manufacturing yield due to statistical deviation of component values of Signal In Artificial intelligence, an evolutionary algorithm (EA is a Subset of Evolutionary computation, a generic population-based Metaheuristic Fitness (often denoted w in Population genetics models is a central concept in evolutionary theory. e. the probability of becoming a parent to new individuals in the population - constant (and without the need for any knowledge about average information as a criterion function). This is illustrated by the figure below, showing Gaussian adaptation climbing a mountain crest in a phenotypic landscape. The lines in the figure are part of a contour line enclosing a region of acceptability in the landscape. At the start the cluster of red points represents a very homogeneous population with small variances in the phenotypes. Evidently, even small environmental changes in the landscape, may cause the process to become extinct.

After a sufficiently large number of generations, the increase in average information may result in the green cluster. Actually, the mean fitness is the same for both red and green cluster (about 65%). Fitness (often denoted w in Population genetics models is a central concept in evolutionary theory. The effect of this adaptation is not very salient in a 2-dimensional case, but in a high-dimensional case, the efficiency of the search process may be increased by many orders of magnitude.

Besides, a Gaussian distribution has the highest average information as compared to other distributions having the same second order moment matrix (Middleton 1960).

But it turns out that this is not in general a good measure of uncertainty or information. For example, the differential entropy can be negative; also it is not invariant under continuous co-ordinate transformations. Jaynes showed in fact in [1] (sect. Jaynes is a Surname, and may refer to Dwight Jaynes, American sports journalist Edwin Thompson Jaynes, American physicist 4b) that the expression above is not the correct limit of the expression for a finite set of probabilities.

The correct expression, appropriate for the continuous case, is the relative entropy of a distribution, defined as the Kullback-Leibler divergence from the distribution to a reference measure m(x),

(or sometimes the negative of this). In Probability theory and Information theory, the Kullback–Leibler divergence (also information divergence, information gain, or relative

The relative entropy carries over directly from discrete to continuous distributions, and is invariant under co-ordinate reparamatrisations. For an application of relative entropy in a quantum information theory setting, see eg [2]. Quantum statistical mechanics is the study of Statistical ensembles of quantum mechanical systems.

Theoretical relationship

Despite all that, there is an important difference between the two quantities. The information entropy H can be calculated for any probability distribution (if the "message" is taken to be that the event i which had probability p_i occurred, out of the space of the events possible). But the thermodynamic entropy S refers to thermodynamic probabilities p_i specifically.

Furthermore, the thermodynamic entropy S is dominated by different arrangements of the system, and in particular its energy, that are possible on a molecular scale. In comparison, information entropy of any macroscopic event is so small as to be completely irrelevant.

However, a connection can be made between the two, if the probabilities in question are the thermodynamic probabilities p_i: the (reduced) Gibbs entropy σ can then be seen as simply the amount of Shannon information needed to define the detailed microscopic state of the system, given its macroscopic description. Or, in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more". Gilbert Newton Lewis ( October 23, 1875 - March 23, 1946) was a famous American physical chemist known for the discovery Year 1930 ( MCMXXX) was a Common year starting on Wednesday (link will display 1930 calendar of the Gregorian calendar. To be more concrete, in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions that need to be answered in order to fully specify the microstate, given that we know the macrostate.

Furthermore, the prescription to find the equilibrium distributions of statistical mechanics, such as the Boltzmann distribution, by maximising the Gibbs entropy subject to appropriate constraints (the Gibbs algorithm), can now be seen as something not unique to thermodynamics, but as a principle of general relevance in all sorts of statistical inference, if it desired to find a maximally uninformative probability distribution, subject to certain constraints on the behaviour of its averages. In Statistical mechanics, the Gibbs algorithm, first introduced by J The principle of maximum entropy is a postulate about a universal feature of any Probability assignment on a given set of Propositions ( Events hypotheses (These perspectives are explored further in the article Maximum entropy thermodynamics). In Physics the Maximum entropy school of thermodynamics (or more colloquially the MaxEnt school of thermodynamics initiated with two papers published in the Physical

Information is physical: (1) Szilard's engine

A neat physical thought-experiment demonstrating how just the possession of information might in principle have thermodynamic consequences was established in 1929 by Szilard, in a refinement of the famous Maxwell's demon scenario. A thought experiment (from the German Gedankenexperiment) is a proposal for an Experiment that would test a Hypothesis or Theory Year 1929 ( MCMXXIX) was a Common year starting on Tuesday (link will display the full calendar of the Gregorian calendar. Leó Szilárd (Szilárd Leó February 11, 1898 – May 30, 1964) was an Hungarian - American Physicist who Maxwell's demon was an 1867 Thought experiment by the Scottish Physicist James Clerk Maxwell, meant to raise questions about the possibility

Consider Maxwell's set-up, but with only a single gas particle in a box. If the supernatural demon knows which half of the box the particle is in (equivalent to a single bit of information), it can close a shutter between the two halves of the box, close a piston unopposed into the empty half of the box, and then extract k_BTln2 joules of useful work if the shutter is opened again. The particle can then be left to isothermally expand back to its original equilibrium occupied volume. In just the right circumstances therefore, the possession of a single bit of Shannon information (a single bit of negentropy in Brillouin's term) really does correspond to a reduction in physical entropy, which theoretically can indeed be parlayed into useful physical work. Negative Entropy or negentropy or syntropy of a living system is the entropy that it exports to maintain its own entropy low (see Entropy and life

Information is physical: (2) Landauer's principle

Main article: Landauer's principle

In fact one can generalise: any information that has a physical representation must somehow be embedded in the statistical mechanical degrees of freedom of a physical system. Landauer's Principle, first argued in 1961 by Rolf Landauer of IBM, holds that "any logically irreversible manipulation of information such as the erasure

Thus, Rolf Landauer argued in 1961, if one were to imagine starting with those degrees of freedom in a thermalised state, there would be a real reduction in thermodynamic entropy if they were then re-set to a known state. Rolf Landauer (1927–1999 was an IBM Physicist who in 1961 demonstrated that when Information is lost in an Irreversible circuit, the Year 1961 ( MCMLXI) was a Common year starting on Sunday (link will display full calendar of the Gregorian calendar. This can only be achieved under information-preserving microscopically deterministic dynamics if the uncertainty is somehow dumped somewhere else — ie if the entropy of the environment (or the non information-bearing degrees of freedom) is increased by at least an equivalent amount, as required by the Second Law, by gaining an appropriate quantity of heat: specifically kT ln 2 of heat for every 1 bit of randomness erased.

On the other hand, Landauer argued, there is no thermodynamic objection to a logically reversible operation potentially being achieved in a physically reversible way in the system. It is only logically irreversible operations — for example, the erasing of a bit to a known state, or the merging of two computation paths — which must be accompanied by a corresponding entropy increase.

Applied to the Maxwell's demon/Szilard engine scenario, this suggests that it might be possible to "read" the state of the particle into a computing apparatus with no entropy cost; but only if the apparatus has already been SET into a known state, rather than being in a thermalised state of uncertainty. To SET (or RESET) the apparatus into this state will cost all the entropy that can be saved by knowing the state of Szilard's particle.

Negentropy

Shannon entropy has been related by physicist Léon Brillouin to a concept sometimes called negentropy. Léon Nicolas Brillouin ( August 7, 1889 &ndash December 1969 was a French physicist Negative Entropy or negentropy or syntropy of a living system is the entropy that it exports to maintain its own entropy low (see Entropy and life In his 1962 book Science and Information Theory, Brillouin described the Negentropy Principle of Information or NPI, the gist of which is that acquiring information about a system’s microstates is associated with a decrease in entropy (work is needed to extract information, erasure leads to increase in thermodynamic entropy). ^[1] There is no violation of the second law of thermodynamics, according to Brillouin, since a reduction in any local system’s thermodynamic entropy results in an increase in thermodynamic entropy elsewhere. Negentropy is a controversial concept as it yields Carnot efficiency higher than one.

Black holes

Stephen Hawking often speaks of the thermodynamic entropy of black holes in terms of their information content. Stephen William Hawking CH, CBE, FRS, FRSA (born 8 January 1942 is a British theoretical physicist. A black hole is a theoretical region of space in which the Gravitational field is so powerful that nothing not even Electromagnetic radiation (e Do black holes destroy information? See Black hole thermodynamics and Black hole information paradox. In Physics, black hole thermodynamics is the area of study that seeks to reconcile the Laws of thermodynamics with the existence of Black hole Event The black hole information paradox results from the combination of Quantum mechanics and General relativity.

Quantum theory

Hirschman showed in 1957, however, that Heisenberg's uncertainty principle can be expressed as a particular lower bound on the sum of the entropies of the observable probability distributions of a particle's position and momentum, when they are expressed in Planck units. In Quantum physics, the Heisenberg uncertainty principle states that locating a particle in a small region of space makes the Momentum of the particle uncertain Planck units are Units of measurement named after the German physicist Max Planck, who first proposed them in 1899 (One could speak of the "joint entropy" of these distributions by considering them independent, but since they are not jointly observable, they cannot be considered as a joint distribution. )

It is well known that a Shannon based definition of information entropy leads in the classical case to the Boltzmann entropy. It is tempting to regard the Von Neumann entropy as the corresponding quantum mechanical definition. But the latter is problematic from quantum information point of view. Consequently Stotland, Pomeransky, Bachmat and Cohen have introduced a new definition of entropy that reflects the inherent uncertainty of quantum mechanical states. This definition allows to distinguish between the minimum uncertainty entropy of pure states, and the excess statistical entropy of mixtures.

• The information entropy of quantum mechanical states, Europhysics Letters 67, 700 (2004).

The fluctuation theorem

The fluctuation theorem provides a mathematical justification of the second law of thermodynamics under these principles, and precisely defines the limitations of the applicability of that law to the microscopic realm of individual particle movements. The fluctuation theorem (FT is a theorem from Statistical mechanics dealing with the relative probability that the Entropy of a system which is currently away from The second law of Thermodynamics is an expression of the universal law of increasing Entropy, stating that the entropy of an Isolated system which

Topics of recent research

Is information quantized?

In 1995, Tim Palmer signalled two unwritten assumptions about Shannon's definition of information that may make it inapplicable as such to quantum mechanics:

• The supposition that there is such a thing as an observable state (for instance the upper face of a die or a coin) before the observation begins

• The fact that knowing this state does not depend on the order in which observations are made (commutativity)

The article Conceptual inadequacy of the Shannon information in quantum measurement [3], published in 2001 by Anton Zeilinger [4] and Caslav Brukner, synthesized and developed these remarks. Year 1995 ( MCMXCV) was a Common year starting on Sunday. Events of 1995 Tim Palmer is a British Music producer, Audio engineer and songwriter of rock and Alternative music. Quantum mechanics is the study of mechanical systems whose dimensions are close to the Atomic scale such as Molecules Atoms Electrons Anton Zeilinger (born on 20 May 1945 in Ried im Innkreis, Austria) is an Austrian quantum physicist The so-called Zeilinger's principle suggests that the quantization observed in QM could be bound to information quantization (one cannot observe less than one bit, and what is not observed is by definition "random").

But these claims remain highly controversial. For a detailed discussion of the applicability of the Shannon information in quantum mechanics and an argument that Zeilinger's principle cannot explain quantization, see Timpson [5] 2003 [6] and also Hall 2000 [7] and Mana 2004 [8], who shows that Brukner and Zeilinger change, in the middle of the calculation in their article, the numerical values of the probabilities needed to compute the Shannon entropy, so that the calculation makes no sense.

For a tutorial on quantum information see [9].

See also

• Thermodynamic entropy
• Information entropy
• Thermodynamics
• Statistical mechanics
• Information theory
• Physical information
• Fluctuation theorem
• Black hole entropy
• Black hole information paradox

External links

• Entropy is Simple...If You Avoid the Briar Patches. In Thermodynamics (a branch of Physics) entropy, symbolized by S, is a measure of the unavailability of a system ’s Energy In Physics, thermodynamics (from the Greek θερμη therme meaning " Heat " and δυναμις dynamis meaning " Statistical mechanics is the application of Probability theory, which includes mathematical tools for dealing with large populations to the field of Mechanics Information theory is a branch of Applied mathematics and Electrical engineering involving the quantification of Information. In Physics, physical information refers generally to the Information that is contained in a Physical system. The fluctuation theorem (FT is a theorem from Statistical mechanics dealing with the relative probability that the Entropy of a system which is currently away from In Physics, black hole thermodynamics is the area of study that seeks to reconcile the Laws of thermodynamics with the existence of Black hole Event The black hole information paradox results from the combination of Quantum mechanics and General relativity. Dismissive of direct link between information-theoretic and thermodynamic entropy.

References

• C. H. Bennett, "Logical reversibility of computation," IBM Journal of Research and Development, vol. 17, no. 6, pp. 525-532, 1973.
• Leon Brillouin, Science and Information Theory, Mineola, N. Y. : Dover, [1956, 1962] 2004. ISBN 0-486-43918-6
• Michael P. Frank, "Physical Limits of Computing", Computing in Science and Engineering, 4(3):16-25, May/June 2002.
• Andreas Greven, Gerhard Keller, and Gerald Warnecke, editors. Entropy, Princeton University Press, 2003. ISBN 0-691-11338-6. (A highly technical collection of writings giving an overview of the concept of entropy as it appears in various disciplines. )
• I. Hirschman, A Note on Entropy, American Journal of Mathematics, 1957.
• R. Landauer, Information is Physical Proc. Workshop on Physics and Computation PhysComp'92 (IEEE Comp. Sci. Press, Los Alamitos, 1993) pp. 1-4.
• R. Landauer, Irreversibility and Heat Generation in the Computing Process IBM J. Res. Develop. Vol. 5, No. 3, 1961
• H. S. Leff and A. F. Rex, Editors, Maxwell's Demon: Entropy, Information, Computing, Princeton University Press, Princeton, NJ (1990). ISBN 0-691-08727-X
• Middleton, D. An Introduction to Statistical Communication Theory. McGraw-Hill, 1960.
• Claude E. Shannon. A Mathematical Theory of Communication. "A Mathematical Theory of Communication" is an influential 1948 article by Mathematician Claude E Bell System Technical Journal, July/October 1948.