Eugene Wigner reasoned that the remarkable “effectiveness of mathematics in the natural sciences” is “unreasonable.” I argue in Nature is Rational that it is reasonable. Here I argue that the remarkable ineffectiveness of Information Theory is unreasonable, given the importance of information to everything.
The Unreasonable Ineffectiveness of Information Theory
Information Theory has been unreasonably ineffective, but it has not been ineffective. It is well known and useful in electrical engineering.
The principles of Information Theory have in fact been more successful in predicting the biophysical properties of the brain than any other set of principles. There have been many beautiful studies that quantitatively relate these principles to the biophysical properties of neurons. This includes my own research (for example, Kim and Fiorillo, 2017). A relatively comprehensive account can be found in Principles of Neural Design by Sterling and Laughlin.
Nonetheless, it is remarkable how little impact Information Theory has had in science, particularly with respect to areas where ‘information’ is presumed to be central, such as neuroscience and artificial intelligence. Its success in neuroscience has been almost entirely restricted to early sensory systems (like the retina). Furthermore, although neuroscientists agree that the purpose of the brain is to “process information,” few of them can specify exactly what they mean by “process” or “information,” let alone how to measure information. Most are not even aware of the neuroscience literature related to Information Theory. I had both a bachelors and PhD degree in neuroscience before I even knew that “information” had any technical and mathematical definition. Information Theory is not taught in biology or psychology or medicine, not even at the graduate level.
If the purpose of the brain is to “process information,” as nearly everyone says, why has Information Theory been so ineffectual? And could its ineffectiveness be telling us something about why the brain still seems so mysterious?
Information Theory Lacks a General Measure of Information
One problem is that what we call “Information Theory” is not a general theory of information. It was invented primarily by Claude Shannon, who originally called it “a mathematical theory of communication.” “Communication” is a much more apt name than “information,” since the theory provides only a measure of information that is communicated (transmitted), meaning information that is gained from observation. The lower the probability of a new observation, the more information is gained from that observation. “Shannon information” is a measure of that information gain, or “new information.”
A major limitation is that the amount of Shannon (new) information is entirely determined by the prior (old) information of the observer, yet Information Theory provides no measure of this prior information. It assumes that there is a prior probability conditional on that prior information, but it provides no general means of identifying that probability (nor does it specify what that probability measures). To the extent that Shannon information measures information, it does not measure old information, or new information, or total information. Rather it measures a relation between old and new information (as does ‘Fisher information‘).
What is needed is a general theory and measure of information that includes “Information Theory” as a special case. This would allow us to measure the total information of an observer, regardless of whether that information is old or new. The first step is to understand what probability measures, since it is more fundamental that information (information is a function of probability). Probability measures evidence, but few people recognize that, as I discuss in The Measure of Evidence.
Information Theory Is from the External Perspective of the Scientist or Engineer
A second major limitation of Information Theory, and the main reason that it has had only a modest impact on neuroscience, is that it implicitly assumes that the observer must be the scientist or engineer. Its stated purpose is to quantify information communicated from a “sender” to a “receiver.” Shannon was particularly interested in the case of a telephone conversation (he was an engineer at Bell Labs). As a neuroscientist I am particularly interested in the case that the sender is the external environment and the receiver is a neuron. Regardless of what the sender and receiver might be, the problem is that the entire theory was formulated from the third-person perspective of an external observer of the sender and receiver.
If Shannon is the external observer, he quantifies how much information he gains about the state of the sender by observing the state of the receiver. The quantity Shannon finds depends entirely on his prior information about the sender and receiver. It has no dependence at all on the prior information of the sender or receiver (though in some special cases sender, receiver, and external observer all share some or all of their prior information). This is not obvious in the literature, at least in part because many people do not recognize the importance and observer-dependence of prior information and probability (Fiorillo, 2012).
Information Theory in Neuroscience
I was introduced to Information Theory in 2002 when I read Spikes: Exploring the Neural Code, by Bill Bialek and colleagues. It is well written, and the importance of the topic was obvious. I was particularly persuaded by the objective, stated at the start of the book, to “take the neuron’s point of view.”
However, I could not entirely make sense of the ideas in Spikes until I read Probability Theory: The Logic of Science by E.T. Jaynes the following year. Then I recognized that the analysis in Spikes, and in all other applications of Information Theory to neurons going back to Barlow in 1961, is from the perspective of the physiologist. Since 2003 I have been determined to actually take the Neuron’s Point of View. This is possible using the “objective Bayesian” theory of probability promoted by Jaynes, as described in The Measure of Evidence.
In conventional applications of Information Theory to neurons, the physiologist is the observer, and it is only the prior information of the physiologist that matters (for example, see Brenner, Bialek, and de Ruyter van Steveninck, 1999). The physiologist uses that prior information to quantify how much information he or she gains about the neuron’s sensory input by observing the neuron’s spike output. In the vast majority of studies, almost no attempt is even made to consider the neuron’s prior information and corresponding point of view.
In some special cases the prior information of the neurons happens to closely match that of the physiologist. For example, this is the case in retinal ganglion neurons and midbrain dopamine neurons. In these special cases we find that the relation between the neuron’s sensory input and its spike output complies beautifully with principles of Information Theory (some otherwise puzzling data that I collected from dopamine neurons suddenly made sense to me once I understood these principles; see Tobler et al 2005).
But the prior information of most neurons does not match that of the physiologists who study them. Since the physiologist does not try to take the neuron’s perspective, by trying to infer what prior information the neuron has, it appears to the physiologist that the typical neuron does not comply with the principles of Information Theory. For example, this is is the case in retinal photoreceptors (the cells that respond directly to light). They are therefore viewed by conventional information theorists as redundant and inefficient in their representation of the visual world.
The Neuron’s Information and Point of View
The solution is to acknowledge that the neuron has its own prior information and corresponding perspective, as I describe in my General Theory of the Brain. There are countless perspectives, but only the neuron’s perspective matters to the neuron’s function. To take its perspective we need a model that specifies what prior information the neuron actually has, and we need to ignore any additional information we have. If we do this carefully it will be found that the principles of Shannon’s Information Theory apply to all neurons (though in a less obvious manner in neurons of motor systems; see Fiorillo, 2010; Fiorillo et al., 2014; Kim et al., 2015).
There is hardly a bigger advance in human evolution than our cognitive ability to imagine the perspective of others. It is a major feature that distinguishes us from almost all other animals. It allowed civilization to develop, and it led directly to major advances in physics, including the Copernican Revolution and Newton’s Mechanics (see Science’s Problem with Observers). If the concept of local observers is so useful in physics, and in our relations with other people, why should it not be just as useful in biology and the theory of information?