Subjective Sense-Making

In the big picture what I'm interested in is a story which describes the evolution of intelligence through transitions in information processing and cycles of creating and exploiting emergent structure in the world; the richer and more abstract the information about the world a cognitive system is able to exploit, the wider its ability to impact the world and solve complex problems. This project of describing the relationship between living systems and emergent structure can be framed as connecting physics and phenomenology - connecting the objective physical reality of the world with how it is experienced and modifed by subjective agents acting in it. The story of mind and life.

Below I begin contextualizing how natural intelligence is fundamentally situated in the world and the product of subjective sense-making evolution developed in service of guiding adaptive behavior. In the first section I begin to frame the problem of intelligence from the subjective perspective and illuminate how cognitive systems are subjective world models. In the second section I introduce Jakob von Uexküll's Umwelt theory to further illustrate the subjective perspective of living systems. In the third section I explicate what it means for cognitive systems to "exploit structure in the world". In the fourth section I introduce James Gibson's theory of affordances as a theoretical concept to help conceptualize subjectively perceived structure and the reciprocity defining the relationship between cognitive systems and the world. In the fifth section I discuss the representational scaffolding and flexibility underlying human intelligence. At the end I will briefly review the discussion and restate the subjective nature of the world models built by evolution and their adaptive purpose.

The subjective world is an amalgamation of features of the outside world that have been captured and integrated into a perspective on the world, the output of a world model engaged in a dance of interpretation and action with the external world. The point here is that the world in experience - from a cognitive system's perspective - is defined by the particular means the system possesses for capturing and processing information about the world. What we know about the world and how we act flows from our sense-making capabilities and the structure we can perceive in the world.

I want to emphasize that despite it being trivially true that our subjective frame follows from our physical organization, it is difficult to really internalize the contingent character of the familiar phenomenological tropes that format our own subjective worlds. Such internalization helps us see the world from different perspectives and appreciate the power and purpose of different representational features and environmental engagements. Our subjective worlds could be different, and are very different from organisms like apes, fish, and ants. The single physical world we all exist within supports countless world models; humans collectively occupy only a small area in the space of all possible minds. However, even within this narrow band there is a wide space of possibilities. One can read the amazing and sometimes tragic literature of abnormal psychology to gain a deeper appreciation for the ways in which our subjective realities could be different given adjustments to our cognitive architectures.

So why focus on the subjective perspective of cognitive systems at all? Because it is the structure of a cognitive systems subjective world which entirely reflects its knowledge of the outside world, ability to integrate new information, and horizon of possible actions. Subjective meaning is the fuel for intelligence as a system can only act in response to signals that have meaning for it. If we want to understand how adaptive systems learn to make sense of the world then we must be able to discuss the content of this sense-making, the world from the system's perspective - the subjective world. The sense of subjectivity we care about here is in the relationship between a system and the world, discussion of consciousness is not required - although adjacent. What matters are the dynamics between an adaptive system and the environment, dynamics that can be described for dogs, plants, and viruses. For humans this relationship is colored in vivid conscious experience, and I started with this picture because it is familiar, but we can just as well consider the structure of the world from the perspective of a single cell or other basic forms of life.

To begin formalizing the study of animals' subjective worlds, von Uexküll used the term umwelt ("environment") to describe the world as it is interpreted by an organism. The umwelt is the perceived world within which a subject acts. It encapsulates all the features of the external world that carry significance to the organism given it's sense-making resources and functional entanglement with the environment. The umwelt is complemented by the umgebung which describes the full external reality.

In addition to championing the subjective nature of living systems, Umwelt theory also emphasizes the role of the environment in cognition. If we do not appreciate the peculiar ways cognitive systems' relate to their environments then we cannot understand the extent of their abilities and intelligence. We shouldn't study fish out of water nor birds in cages, these artificial environments being far removed from the environmental niches in which they are built to act well in. The environment is a crucial partner in the evolution of adaptive behavior. Adaptive systems do not learn high fidelity representations of the world for their own sake, but instead learn representations and habits that guide effective behavior in their local environments. This recognition of the importance of the environment sets the table nicely for further exploration of distributed intelligence and theories that extend cognition beyond the narrow confines of the skull.

What we get here is a framing of adaptive behavior that focuses on the subjective sense-making of living systems and their distinctive entanglements with the environment. These entanglements are feedback loops relating meaningful features of the world to possible actions. The umwelt isolates all the information a creature can interact with. It describes a world model, and highlights the subjective and action-oriented nature of biological intelligence. These are nice footholds when considering how adaptive systems learn rich models of the world.

But what does it actually mean to "exploit structure" as I've been causally saying? The phrase is shorthand for "capture some information about the world, process it, and take advantage of it in continued engagement with the world". Exploiting structure entails using some gained knowledge about the real world to achieve some end. I have loosely used the phrase to describe individuals acting in the world as well as learning that occurs across generations. For example, a simple prokaryote exploits information about its surrounding chemical gradient in navigating the world; in fact, the chemical gradient exhausts the information about the world it is able to exploit. On the other end of the spectrum are humans, whose evolutionary inheritance is a genetic code within which evolution has compressed an immense amount of information about the world into. The result of this genetic compression of information about the world is that we are built to exploit tons of structure in the world, have rich ways of understanding the world, and are thus pretty intelligent systems.

By exploiting the structure of physics, biology creates emergent levels of structure and complexity. The dynamics governing mechanical physics at the biological level allow for objects with particular shapes and characteristics to capture, process, and integrate specific information about the world. The building blocks of biological life are able to store and direct energy using movement patterns that exploit the dynamics of the mechanical force. The state of these objects relates to the real world and this information can be used to do different types of work. Shapes that persist through time and direct energy in useful ways can evolve. As these basic self-organizing systems evolve, the range of possible collective computations grows, allowing for more complex organization and new functional engagements with the real world.

In building world models to process and exploit structure in the world, evolution discovered ways of capturing and using information about the world that are particularly helpful for guiding adaptive behavior. Chemical-sensing, epigenetics, nervous systems, linguistic communication - these are all major transitions in information processing that fueled adaptive success for their implementers by broadening their horizons of perception and action. More complex world models make use of information that was previously lost to noise to do causal work on behalf of the subject perceiving the world. As evolution naively explores possibilities and innovates in response to adaptive challenges, opportunities for increased complexity and intelligence appear.

Keep in mind the subjective frame here as sense-making serves a subjective perspective and all adaptations necessarily exploit some accessible structure in the world to enable productive behaviors. The exploited structure could be at the level of fundamental physics, chemical gradients, or social dynamics. The simplest illustration of this is the above example of the emergent mechanics of basic biological structures. Trivially, all living systems exploit the laws of physics to maintain themselves. However, this framing is just as useful in describing complex systems exploiting higher-level domains to accomplish goals. Higher-level structure plays a similar role in framing an agent's world as the mechanical force does for simple biological objects. Although, the dynamics between physical systems and emergent higher-levels of structure are more opaque. Next I'll introduce the concept of affordances to help us better frame a cognitive systems relation to the world.

So what's an affordance? The term has been adopted and used by different communities in varying ways, particularly in the field of human-computer interfaces and in design generally. At their core, affordances are an abstract category of behavioral opportunities defined by the possibilities an environment offers to a particular agent. For most humans a chair affords sitting, a pencil affords writing, and wood affords fire building. For an ant these three affordances do not exist. Affordances manifest at the intersection of a cognitive system's action dispositions given its internal constitution and the reality of the environment the system inhabits. Affordances describe a domain of structure in the world from a subjective perspective, and really in a sense capture the structure of the world for a particular cognitive system. To perceive an affordance is to perceive a possibility. There are traces of von Uexküll's ideas here in the agent relative status of affordances and the focus on action-oriented engagement with environmental niches; the umwelt is colored by affordances.

Perhaps in contrast to strict Gibsonians, I prefer a wider notion of affordances that extends above and below the level of single organism of cat or human for which we usually reserve the term perception. I find the extended concept to be a useful companion in considering how adaptive systems evolve in relation to the structure of the world generally. From a system's subjective perspective the world always offers a particular suite of possibilities, and this is as true for simple biological objects as for an entire organism. Zooming out over longer time scales we can consider evolutionary affordances, the possible evolutionary paths, present for particular physical systems in certain environmental niches.

Because the set of all possible affordances for a system entirely reflects how a system perceives and can interact with the world, the solution space for an adaptive problem is necessarily a subset of a system's set of possible affordances. Over time, adaptive systems learn to exploit, by internalizing, the structure upon which evolutionarily rewarding affordances rest - the result being a more adaptive set of affordances. This internalization describes developments from the physical organization cells and nervous systems take on, to the emergence of social cooperation among humans. A clear example are the visual priors that visual systems learn which mirror the statistics of the surrounding visual world. This is why the many visual illusions we see today are built with patterns and compositions that are rare, if at all present, in the natural world. Adaptions reflect learned world knowledge. As systems adapt and expand their world models to exploit accessible domains of structure in the world such as chemical, visual, or social information, they create new representations and affordances. Evolution is an effective, although inefficient, search mechanism for learning about the structure of the world.

If we return to the general notion of affordances as the possible actions offered by the environment to an agent, we see that affordances also clearly emphasize the situated nature of cognitive systems. Affordance theory tells us we perceive things in relation to the possibilities they present for action, which themselves describe the coupled dynamics between an agent and the environment; natural intelligence is fundamentally situated in the world. By giving us a way to conceive of the subjective meaning of a signal to a cognitive system, affordances are a powerful concept that are useful for a wide variety of questions about cognition and intelligence.

The common representational concepts that define umwelt across the biosphere are prominent because they pick out information about the world that has proven effective for adaptive behavior. Over time these useful ways of relating to the world were grooved into living systems. Similar to how basic biological objects converge on certain shapes and mechanics for doing work, cognitive systems on Earth converge on basic representational constructs for understanding the world. Many of these core representational concepts like embodiment, space, objects, and intuitive physics frame our own subjective worlds and are so familiar that they can be hard to appreciate as important components of intelligence.

For example, it is hard to imagine living without an intuitive concept for something like the general category of object, a subjective world where things like rocks and fire sit idle outside one's cognitive loops, registering zero affordances. Instead, we are equipped to flexibly integrate new information and emergent possibilities. When I encounter new objects I already know a lot of information about them given the web of world knowledge I already have. For example: I know they will fall when dropped unless specifically designed otherwise, that sharp objects may cut me, and that heavy objects are hard to move. These basic intuitions betray a vast amount of world knowledge that is constantly helping me navigate the world. We take our knowledge about the world for granted because it simply is the world for us as it appears in our subjective experience.

We have been tuned to the world we live in by evolution. Our subjective worlds are filled with structure and affordances that support our behavior and the preservation of our informational integrity. The rich representational scaffolding we have for making sense of the world forms the stage upon which new and higher-level entities gain meaning. We seamlessly find meaning in language, art, religion, social relationships, mathematics, and culture generally because of the rich forms of sense-making we inherited from evolution. Human intelligence is so far unique on Earth, but the dynamics responsible for our intelligence and meaning-making extend throughout all of evolution. In trying to understand the development of the basic representational concepts underlying our own general intelligence we should keep in mind the subjective and action-oriented framework within which these competencies emerged.