What is the relationship between information, causation, and entropy?

The other day, I was reading a post from Corey S. Powell on how we are all ripples of information. I found it interesting because it resonated with my own understanding of information (i.e. it flattered my biases). We both seem to see information as something active rather than passive. In my case I see it fundamentally related to causation itself, more specifically a snapshot of causal processing. Powell notes that Seth Lloyd has an excellent book on this topic, so I looked it up.

Lloyd’s 2006 book is called Programming the Universe, which by itself gives you an idea of his views. He sees the entire universe as a giant computer, specifically a quantum computer, and much of the book is about making a case for it. It’s similar to the “it from qubit” stance David Chalmers explores in his book Reality+. (I did a series of posts on Chalmers’ book a while back.)

One of the problems with saying the universe is a computer is it invites an endless metaphysical debate, along with narrow conceptions of “computer” leading people to ask things like what kind of hardware the universe might be running on. I’ve come to think a better strategy is to talk about the nature of computation itself. Then we can compare and contrast that nature with the universe’s overall nature, at least to the extent we understand it.

Along those lines, Chalmers argues that computers are causation machines. I think it helps to clarify that we’re talking about logical processing, which is broader than just calculation. I see logical processing as distilled causation, specifically a high degree of causal differentiation (information) at the lowest energy levels currently achievable, in other words, a high information to energy ratio.

The energy point is important, because high causal differentiation tends to be expensive in terms of energy. (Data centers are becoming a major source of energy consumption in the developed world, and although the brain is far more efficient, it’s still the most expensive organ in the body, at least for humans.)

Which is why computational systems always have input/output interfaces that reduce the energy levels of incoming effects from the environment to the levels of their internal processing, and amplify the energy of outgoing effects. (Think keyboards and screens for traditional PCs, or sense organs and muscles for nervous systems.)

Of course, there’s no bright line, no sharp threshold in the information / energy ratio where a system is suddenly doing computation. As a recent Quanta piece pointed out, computation is everywhere. But for most things, like stars, the magnitude of their energy level plays a much larger role in the causal effects on the environment than their differentiation.

However, people like Lloyd or Chalmers would likely point out that the energy magnitude is itself a number, a piece of information, one that has computational effects on other systems. In a simulation of that system, the simulation wouldn’t have the same causal effects on other physical systems as the original, but it would within the environment of the simulation. (Simulated wetness isn’t wet, except for entities in the simulation.)

Anyway, the thing that really caught my eye with Lloyd was his description of entropy. I’ve covered before my struggles with the customary description of entropy as the amount of disorder in a system. Disorder according to who? As usually described, it leaves the question of how much entropy a particular system has as observer dependent, which seems problematic for a fundamental physics concept. My reconciliation of this is to think of entropy as disorder for transformation, or in engineering terms: for work.

Another struggle has been the relationship between entropy and information. I’ve long wanted to say that entropy and information are closely related, if not the same thing. That seems like the lesson from Claude Shannon’s theory of information, which uses an equation similar to Ludwig Boltzmann’s for entropy. Entropy is a measure of the complexity in a system, and higher values result in a system’s energy gradients being fragmented, making much of the energy in the system unavailable for transformation (work), at least without adding additional energy into the system.

However, people like Sean Carroll often argue that a high entropy state is one of low information. Although Carroll does frequently note that there are several conceptions of “information” out there. His response makes sense for what is often called “semantic information”, that is information whose meaning is known and useful to some kind of agent. The equivalence seems more for “physical information”, the broader concept of information as generally used in physics (and causes hand wringing due to the possibility of black holes losing it).

Lloyd seems to be on the same page. He sees entropy as information, although he stipulates that it’s hidden information, or unavailable information (similar to how energy is present but unavailable). But this again seems to result in entropy being observer dependent. If the information is available to you but not me, does that mean the system has higher entropy for me than it does for you? If so, then computers are high entropy systems since none of us have access to most of the current information in the device you’re using right now.

My reconciliation here is to include the observer as part of the accounting. So if a system is in a highly complex state, one you understand but I don’t, then the entropy for the you + system under consideration is lower than the entropy for the me + system combo. In other words, your knowledge, the correlations between you and the system, makes the combined you + system more ordered for transformation than the me + system combo. At least that’s my current conclusion.

But that means for any particular system considered in isolation, the level of entropy is basically the amount of complexity, of physical information it contains. That implies that the ratio I was talking about above, of information to energy, is also of entropy to energy. And another way to refer to these computational systems, in addition to information processing systems, is as entropy processing systems, or entropy transformers.

This might seem powerfully counter intuitive because we’re taught to think of entropy as bad. Computational systems seem to be about harnessing their entropy, their complexity, and making use of it. And we have to remember that these aren’t closed systems. As noted above, they’re systems that require a lot of inbound energy. It’s that supply of energy that enables transformation of their highly entropic states. (It’s worth noting that these systems also produce a lot of additional entropy that requires energy to be removed, such as waste heat or metabolic waste.)

So computers are causation machines and entropy transformers. Which kind of sounds like the universe, but maybe in a very concentrated form. Viewing it this way keeps us more aware of the causal relations not yet captured by current conventional computers. And the energy requirements remind us that computation may be everywhere, but the useful versions only seem to come about from extensive evolution or engineering. As Chalmers notes in his book, highly computational systems don’t come cheap.

What do you think? Are there differences between physical information and entropy that I’m overlooking? And how would you characterize the nature of computation? Does a star, rock, or hurricane compute in any meaningful sense? What about a unicellular organism?

Featured image credit

https://selfawarepatterns.com/2024/07/28/entropy-transformers/

I think the brain is a computational system and what we generally refer to as the mind and consciousness are some of its computations. But I’m also aware that the brain works very differently from how a typical digital computer works. One criticism of computationalism that I have some sympathy with is the word “computation” can lead people to think about neural processing in the wrong way.

In technological computing, we generally have a distinction between code and data, between processing and storage. We write programs, lists of instructions, that operate on that data. Of course we often forget that code is itself data. (Which causes a lot of security issues.)

But even at the hardware level, there is a distinction. Most of the computing device you’re using to read this is dedicated to storage: memory chips, SSD storage, etc. The actual doers of the system, the processors, actually make up a small (but expensive) portion. This works because the system has the ability to accurately copy data from storage into processor registers, act on them, and then copy the results back into storage, the benefits of a discrete digital system.

But that’s not how neural networks work. Instead, data and processing are together. We can insist that the strength of synaptic connections count as storage, but since they mediate signaling between neurons, they also count as a kind of code. So the brain, being a physical neural network, doesn’t seem to store information and then act on it. The storage and the acting are one and the same.

What that means is that talk of data being moved, sent, or copied between brain regions is probably a shaky metaphor at best. Instead, some regions react to incoming signals in certain ways, propagate those reactions in signals to other regions, which react in their own way. We can think of the sensory regions as the early reactions and the later higher order regions as reacting to the earlier reactions.

Some of those reactions signal back to the earlier regions, creating recurrent signaling loops. And eventually some of the signaling reactions reach motor regions, which cascade down efferent connections back to muscles, glands, etc. Of course this is what brains are for, to figure out what the best course of action is for the organism.

People often scoff at the idea of a grandmother or Jennifer Aniston neuron, but under this view, it’s more plausible than it might seem. It’s important to remember that this isn’t the only neuron that fires with these concepts, just that the convergence of many other reactions (a human, a woman, certain hair color, certain facial features, certain behaviors, etc). It should be seen as the very tip of a vast reactional iceberg.

And the convergence point is probably different today than it was years ago, as the concept of grandmother or Jennifer Aniston, and what it means for us, changes over time. These changes, called “representational drift“, concern some theorists. But to me they’re only a concern for someone caught up in thinking of a neural representation as data in the same sense as it would be in a technological computer. Once we realize it isn’t, that it’s a set of reactive dispositions, dispositions which can change over time as new things are learned, the drift makes a certain kind of inevitable sense.

This unity of data and processing is why I think understanding information as causation works better than most people realize. The most common reactions against it seem to stem from the idea that causation is action while information itself is inert. But that’s working from the distinction used by current technology. Evolution doesn’t seem to have ever used or cared about that distinction.

Consider what this means for the description of a leading theory of consciousness: global workspace theory. It’s often described in a manner that makes it sound like information in the brain needs to get into a certain location, and once in that location that it’s broadcast throughout the brain. I’ve used that language myself. It’s natural to fall into since it quickly gets the idea across. But it sets up a picture that is arguably misleading, leading some theorists to create models based on that picture rather than the messier reality.

Consider instead a system, a network of neural networks. The sensory portion of that network reacts in certain ways to stimuli. The various reactions all try to propagate. The signal forward is an attempt to create a circuit of reactions. The signals to the sides, to the other reactions happening in parallel, is inhibitory (lateral inhibition). In other words, we have a series of reactions creating multiple parallel signaling circuits, each trying to propagate while inhibiting the others.

As the signals propagate into the network of neural networks, creating ever more elaborate circuits, some of those circuits excite the same regions, which in turn signal back exciting the contributing circuits even more. This leads to coalitions of circuits, and competition between those coalitions, each trying to build themselves while inhibiting the other coalitions.

In standard global workspace theories, eventually one of the coalitions crosses a phase shift boundary and becomes omni-dominant. Its reactions have “entered the global workspace”. In closely related theories, like Daniel Dennett’s multiple draft model, the final resolution is less clear cut. No coalition ever wins completely. It’s the coalition that manages to excite the language and episodic memory centers in certain ways that are retroactively considered to be conscious.

But in both cases, it’s the coalition of circuits which end up dominating the system that become reportable and memorable, that essentially become perceived as part of the stream of consciousness. So alternate names for global workspace could be “global coalition of circuits” or “global coalition of dispositions”, while Dennett’s could be called the “multiple coalitions model”.

Understanding this unity of data and processing also makes it more clear why the idea of everything being piped to one control center is misguided (at least a control center other than the brain overall). Different mental content is different parts of the network reacting, not any one central part reacting to the data. Of course, it doesn’t feel like that, but that’s because we can’t introspect the parts.

Unless of course I’m missing something?

Featured image source

https://selfawarepatterns.com/2023/10/21/the-unity-of-storage-and-processing-in-nervous-systems/