Entropy, and the Metaphor

to ancient languages for the names of important scientific quantities, so that they mean the same thing in all living tongues. I propose, accordingly, to call S the entropy of a body, after the Greek word transformation. I have designedly coined the word entropy to be similar to energy, for these two quantities are so analogous in their physical significance that an analogy of

31 | P a g e

denominations seems to be helpful’. Ben Naim (2009) continues with reference to Tribus’s (1971) story on Shannon’s naming:

‘What’s in a name? In the case of Shannon’s measure the naming was not accidental. In 1961 Tribus asked Shannon what he had thought about when he had finally confirmed his famous measure. Shannon replied, ‘My greatest concern was what to call it. I thought of calling it information, but the word was overly used, so I decided to call it uncertainty. When I discussed this with John von Neumann, he had a better idea. Von Neumann told me: you should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name. In the second place, and more important, no one knows what entropy really is, so in the debate you will always have the advantage’. Tsallis, on the first page of the preface to his 2009 book ‘Introduction to non-extensive statistical mechanics’ quotes the majority of the same text, this time referencing back to Tribus and McIrvine (1971) (readers interested in the further debate on this anecdote should refer to http://www.eoht.info/page/Neumann-Shannon+anecdote). Ben-Naim (2009) goes on to quote Denbigh (1981): ‘In my view von Neumann did science a disservice. There are, of course, good mathematical reasons why information theory and statistical mechanics both require functions having the same formal structure. They have a common origin in probability theory, and they also need to satisfy certain common requirements such as additivity. Yet, this formal similarity does not imply that the functions necessarily signify or represent the same concept. The term ‘entropy’ had already been given a well-established physical meaning in thermodynamics, and it remains to be seen under what conditions, if any, thermodynamic entropy and information are mutually inconvertible’.

It seems, from this anecdote, that the metaphor applied to information theory was somewhat arbitrarily allocated on the basis that the formula looked the same and that there was

32 | P a g e

already some confusion as to what entropy actually is. In addition, previous research on the use of entropy in operations and supply chain management has largely chosen to assume Shannon to be correct in using the term, and form, entropy; thus, these arguments have been developed from the Shannon assumption forward. This research intends to go beneath this assumption, to understand more of the function of the form in order to understand it’s applicability in the business context used herein.

A second point is this, as we will discover later, entropy has been variously linked with degrees of information, uncertainty, mixed up ness etc. However, from an information theoretic perspective a simple description developed by both Gliek (2011) and Ben-Naim (2008) is this: It is the number of additional binary question needed to understand the state of the system. This is easily understood by example. Take 32 identical boxes and into one place an arbitrary object, mix the boxes up. It will take five binary questions to identify the box in which the arbitrary object resides (keep dividing the boxes in half). Another way of describing this is that it will take 𝑙𝑜𝑔232 = 5 binary question. In addition, entropy is a function of volume; as volume increases so too does entropy, the thermodynamic term ‘extensive’ will explain this phenomena later in this section. Later, the text will demonstrate how this assertion misses a point on information structure, which calls for an additional dimension to be added to the assertion for it to remain valid; however, for now, the combination of these two points is the starting point for this research.

A further characteristic of the metaphor pertains to the use of probability and information. Entropy may, for now, reflect – as described above – the number of binary question required. However, this assertion does assumes the role of the experimenter to be the only role in the experiment. A fair coin has entropy S of 𝑝 log2𝑝 = 1. Once the coin is tossed entropy

33 | P a g e

equals zero in the eyes of the experimenter and observers; however, to those who are unaware of the outcome – the information available from the experiment – the entropy - remains at one. This is further demonstrated in the classic thought experiment by Erwin Schrödinger: Outside the box the observer perceives a maximum entropy of one until the box is opened and the state of the cat known. Inside the box, the cat is pretty sure of its state throughout the experiment. The point is that to the observer not directly connected with the experimental activity, there is some form of hidden information that increases entropy. The concept of hidden information will be developed later.

Supply chains are getting ever larger, interconnected, mixed up, etc., generally for all the right reasons; to satisfy an ever more demanding customer. Organisation and governance structures are becoming increasingly complex in order to control the ever more complicated operations. As this volume of process and complexity increases, so to must entropy, being and extensive property (we are assuming the metaphor holds for the time being). Intuitively it seems that a measure of entropy would then provide an understanding of the state of the business that is not available using standard - macro level – measures or language; that is, entropy would provide a measure of the number of questions needed to understand the real state of the business. We will start by reviewing the origins of the term from a statistical mechanics and information perspective, I should point out that the intention is not to critique the formulation of the mathematical constructs except where the formulation is contributing to an understanding of the developing argument. The intention is to review the mathematical construct and the meaning of the term such that we have either a clearly defined understanding, or we have introduced a new term to fit the attributes and characteristics of the thing being measured.

34 | P a g e

It is important to clarify meanings applied through the rest of this thesis. Entropy will refer to the classic version of the form developed by Boltzmann (thermodynamics) and Shannon (Information theory). Other terms will be used throughout this thesis; for instance, ‘missing information’ or ‘hidden information’. Where these other terms are used, the relationship between the other term and entropy will be explained. Also, from a notation perspective, generally 𝑆 is used to denote thermodynamic entropy and 𝐻 for the information theory variant. Both of these notations will be used in this thesis to distinguish the type of entropy being discussed; however, practically, for the purposes of this research the two notations can be considered interchangeable. In some sections reference will be made to different forms of entropy, Renyi entropy, for instance; in these cases the form will be clearly stated.

The term entropy has been attached to the concept of diffusion or complexity as explained by Shannon. The metaphor has been attached because the formula for this diffusion or complexity closely follows the formula for entropy in the thermodynamics environment. This may be appropriate; that is, it is correct to attach the metaphor to the information theoretic formulation because the derivation of the formula in the information theory environment follows a similar logic to that in the thermodynamics environment, they are both based on probability theory. The purpose of the next section is to test this assumption and agree or disagree with the attachment of the metaphor to the common formula. The section is intended to explain the concept from an understanding perspective rather than a perspective of testing the proof for the formulaic development.

2.3.2 The Basics of a Classic Thermodynamic systems. The purpose of this section is to