A note from 2026: the last section of this piece, on how anything can represent everything given the right decoding scheme, is the question my later work on Consciousness and Unambiguous Representations grew out of.

If you have spent any time thinking about consciousness and its relation to brain activity, I’m guessing that the following claim should not sound too unreasonable: The neural correlate of a percept, such as seeing an apple, will at least in some form involve a representation of that apple. This, however, raises many questions: What kinds of representations are required for conscious content? Is the presence of representations enough for phenomenal experience? Are all representations subjectively experienced? A recent paper by Cyriel M.A. Pennartz, introducing Neurorepresentationalism as a new theory of consciousness [1], proposes answers to some of these questions.

In this article, I discuss a few ideas outlined in the paper and embed them within the philosophical context of representationalism and the hard problem of consciousness. Additionally, I introduce a thought experiment that deals with some pretty wild (but speculative) implications of representational accounts of consciousness, and how those might in turn shape our assumptions about the necessary ingredients of subjective experience.

Representationalism

The idea that conscious experiences are intimately connected to mental representations in our brain is not new. Representational theories of consciousness propose that representations in our brain at least partially account for consciousness [2].

One of the types of ‘problems’ that representationalist theories are trying to address is illustrated by the following thought experiment: Say I have a visual percept of an apple on a table, but there is no apple on the table. In other words, I am hallucinating an apple. This raises the question of why I am aware of something apple-like. If the apple is not on the table, where is it? Is it my head? There is definitely no apple in my head, so the apple is nowhere. At first glance this might seem like a meaningless train of thought (at least I thought so at first). Of course there is no apple in my head and of course it’s possible to still see one, as long as the right brain activity is present. However, if there is no actual apple, where does the apple-likeness in my percept come from? Why does this particular neural activity lead to an apple percept? After all, it’s not like neurons in my head are somehow arranged into an apple shape.

This is where representationalism enters the scene: The neurons themselves are not apple-shaped, but what they represent is apple-shaped. This is analogous to the bits of a JPEG binary not being apple-shaped themselves, but they’re encoding an apple-shaped image. This means that, whether the apple is actually on the table or not, the representational content of visual cortex neurons includes that apple.

However, this still leaves open many questions. One of the most pressing ones is the question of what the required attributes of a representation are in order for it to be conscious. Here there are multiple varieties of representationalist theories of consciousness:

PureAny representation constitutes a conscious content.
StrongThe right type of representation is sufficient for conscious contents.
WeakRepresentations are necessary but not sufficient for conscious content.
Types of representationalist views on consciousness, as described in [2].

As far as I understand, Neurorepresentationalism is a member of the strong representationalist theories of consciousness, but more on that in the next section.

For now, let me just say that, even if we decide on which type of representationalism might be true, another question that remains is how the representational content of any neural activity might be decided. To get back to the JPEG example, the same binary digits could represent any arbitrary image given another encoding scheme. Part of this question might be answered by Neurorepresentationalism, but I hold that some mystery remains with any strong representationalist theory, as I will discuss in the final section.

Finally, I doubt that any strong representationalist theory of consciousness will convince everyone that the hard problem is solved. But this might not be the right goal anyways. On the other hand, progress towards solving the Real Problem and maybe even the Core Problem is conceivable (see my article on the hard, real and core problems of consciousness).

Neurorepresentationalism

Having set the stage for the abstract family of representational theories of consciousness, I will now introduce a concrete, fairly recent, instantiation of this group: Neurorepresentationalism [1].

Definition of Consciousness

Motivated by the characterization of essential features of phenomenal consciousness and by its postulated functional role in supporting goal-directed behavior (check out the paper for more details [1]), Pennartz puts forward the following definition:

[Consciousness] is the multi-modally rich, dynamic survey of the subject’s current situation, including his own body and functionally earmarked for planned behavioral and cognitive actions in the future.

My first thoughts when reading a definition like this are: Is this a functional account of consciousness? If so, aren’t we just staying in the realm of the easy problems of consciousness [3] without making any progress on phenomenal consciousness? Pennartz argues that, no, in addition to function, this also addresses phenomenal / qualitative consciousness. He claims that, although consciousness subserves goal-directed behavior, it is not identical to it. In other words, although consciousness is functional with respect to creating a situational multi-modal survey, it is not functional with respect to the overall behavior of the agent. As a result, in terms of input and output, a functionally equivalent agent without consciousness (a functional zombie) is possible. In short, this definition attempts to capture the phenomenal aspect of consciousness without relegating it to a mere epiphenomenon.

Neural Correlate of Consciousness

It should come as no surprise that the proposed correlate of consciousness is a type of neural representation. More concretely, Pennartz proposes conscious representations to be organized in multiple levels of abstraction, each identifiable with constituents of our phenomenal experience at different granularities [1]:

  1. single neurons, having the capacity to respond to single features
  2. ensembles of neurons, forming small, within-area local networks capable of pattern coding within a single submodality (e.g. shape)
  3. unimodal metanetworks, which combine the hypotheses from lower-order ensembles into representations of objects considered within a single modality (e.g. all visual features making up a visual object)
  4. multimodal metanetworks, integrating the information coded by unimodal meta-networks into multisensory object representations

Pennartz illustrates these levels with a diagram of a dinner table in his paper [1].

Pennartz claims that consciousness arises at the highest level of this representational hierarchy, subsuming multiple perceived objects in different modalities. Moreover, these representations are implemented on the neuronal level in such a way that both low-level and high-level cortical areas contribute to conscious contents.

At first glance, it might sound like the previous two statements are conflicting with each other. How can only the highest level of representations be conscious while low-level cortical areas are also involved in consciousness? While tempting, it would be a mistake to equate the representational hierarchy as shown in that diagram with the cortical hierarchy. The important difference is that higher representational levels are not just affected by lower levels, but they subsume the lower levels. Pennartz suggests that, instead of thinking about representations as serially being passed upward in the cortical hierarchy, we should conceive of a representation encompassing multiple cortical areas at once, being ‘vertically integrated’ if you will. Thus, a particular neural assembly can be seen as representing a low-level feature like shape in isolation, while also participating in a higher-level, multi-modal representation of an apple. Note that, because we are identifying consciousness with a certain type of high-level representations, this is an instance of a strong representational theory of consciousness [2].

A further claim is made about the minimal neural substrate for a subjective experience, namely, that it needs to integrate more than one modality. This is also implied by the previously given definition of consciousness as being a ‘multi-modally rich, dynamic survey’. On the face of it, however, this requirement might seem arbitrary. Why couldn’t, for example, the combined system of V1 and V4 create an experience of color perception on its own, without another visual sub-modality having to be part of it? To get to a potential answer to this question, let’s try to take the perspective of V4 itself. V1 (and potentially other areas, I’m no expert on the visual system) projects to V4 and sends electrical signals relevant to color. But V4 doesn’t ‘know’ that the incoming signals relate to color, they could be about anything! Pennartz (I think) argues that, color has its specific character to us precisely because within our conscious experience, it is contrasted to other modalities. The way in which V4 activity correlates with, say, activity in MT, is what gives it its character. This idea reminds me of the information axiom in IIT [4], which asserts that an experience is informative by virtue of excluding other possible experiences. Similarly, the experience might be color-like, by virtue of how it differs from (and correlates to) representations of motion in the visual field.

Given this proposed neural correlate of consciousness, Neurorepresentationalism predicts subjective experience to mainly arise in sensory and associative cortices. This is consistent with claims made about the posterior hot zone as described by Christoph Koch [5].

Multi-Level Representations and the Hard Problem

So far I have introduced a definition of consciousness, which is postulated to capture phenomenal consciousness, and its ostensible neural correlate.

Let’s say that Neurorepresentationalism’s claims about the neural correlate of consciousness are true: Every aspect of our conscious experience can be traced to a multi-level, multi-modal representation within our sensory and associative cortices. Moreover, let’s assume that these representations are not only sufficient, but necessary, i.e. there are no other ways for a system to experience phenomenal consciousness.

In the following, I want to discuss to what extent this would ‘solve’ the problem of consciousness. To approach this, I’m going to make use of three versions of the problem statement: the real problem, the hard problem, and the core problem of consciousness. I describe these in more detail here.

The Real Problem of Consciousness. Anil Seth describes the problem of explaining consciousness as the task of devising a theory, which reliably explains, predicts, and controls conscious experience as a function of brain activity. ‘Explaining’ in this case entails more than just creating a catalogue of neural activity for every possible experience. Instead, we want to find neural mechanisms that are responsible for aspects that are part of our experience. Testing such an explanatory framework can be carried out by validating its predictions against subjects’ verbal reports, or by trying to change neural activity to arrive at a desired phenomenal state. Assuming that Neurorepresentationalism is true, it would fulfill all of these requirements for explaining consciousness. Given a detailed account of how multi-modal, multi-level representations are structured, we can easily map every aspect of our experience to neural mechanism, predict what experience might result from a given representation, and change representations in order to elicit the desired experience. So far so good. However, Chalmers might say that, while at this point we have found a systematic correlate of consciousness, this might not truly explain why or how consciousness arises from this neural activity.

The Hard Problem of Consciousness. One reason we might not be satisfied with a solution that solves the real problem, is because, strictly speaking, we are developing a theory of behavior, namely, verbal reports (or whichever behavior we use as ground truth for conscious experience). In that sense, we are only answering easy problems. Moreover, even if verbal reports are accurate, there would be an explanatory gap of why and how such behavior would be accompanied by phenomenal experience, which rests on conceivability arguments of philosophical zombies. The way the hard problem is framed almost by definition precludes any possible solution to it. This is, as far as I understand, at least partly intentional. I think Chalmers wants us to question our underlying premises (i.e. physical primacy) when trying to solve the problem of consciousness. But that still leaves us with no clear problem statement of what it is we’re trying to find out. The real problem is a good step into the direction of a pragmatic scientific aim for studying consciousness, but it doesn’t address ontological considerations about consciousness, except by (as far as I can tell) making a big implicit assumption about qualia being reducible to physics. Neurorepresentationalism, however, has more to say about this.

The Core Problem of Consciousness. The idea behind this formulation is to acknowledge the existence of both phenomenal consciousness and physical reality, while remaining agnostic as to their exact relation. This leaves us with the following problem statement: We need to explain how the fact that there is something it is like to be us relates to physical matter [6]. This perspective allows for the pragmatic approach suggested by the real problem, while leaving open the question about the metaphysical relationship between consciousness and physics. This leaves room for Pennartz’ statement that

Neurorepresentationalism acknowledges phenomenal experience as real and existing, on an equal ontological footing with the electrophysiological mechanisms ultimately underlying it.

Here I want to make clear that my interpretation of this statement might not reflect the author’s view on the matter exactly. This is because to me there seems to be some friction between ‘on equal ontological footing’ and ‘ultimately underlying it’. I will interpret this passage as saying that electrophysiological data is not more fundamental than phenomenal experience, both are equally real and one cannot exist without the other. However, even if the author intends to say that phenomenal experience is real, but reducible (i.e. conservative realism or weak illusionism as defined by Keith Frankish [7]), a lot of the following discussion is still relevant.

If we simply assume that our phenomenal experience exists, and neurons exist (and all the underlying physics), this still leaves an important question: How can two such seemingly incommensurable things like neurons and experience relate to each other? Pennartz attempts to explain the hardness of reconciling consciousness and neural activity like this: We know a lot about our conscious experience (you could argue it might be the thing we are most familiar with). Moreover, we also know a lot about neurons, and we have an idea about how a neuron of some animal fires upon presenting the right stimulus. However, we know nothing about all the intermediate levels of representations between single neurons and the top-level representation, which is identical to our consciousness (remember, our consciousness is equal to the highest abstraction level of a multi-level, multi-modal representation, the representational content of individual neurons make up the lowest level).

While this explanation provides a tentative account for the hardness of the hard problem, I think someone like Chalmers might still be unsatisfied. The question of why it is that at the highest level, multi-modal representations are associated with phenomenal experience could be seen as unresolved.

This is why I think the core problem might be a better framing, which is to say, instead of answering how/why subjective experience arises from the physical, let’s try to figure out what the relation between the two is. From this perspective, Neurorepresentationalism provides a coherent answer: Phenomenal consciousness relates to the physical by corresponding to the highest level of a hierarchically constructed representation, which at its bottom layer consists of individual (physical) neurons.

Neurons arranged in a certain way imply phenomenal consciousness (by means of forming a representation) just as much as consciousness implies an implementation of a multi-level, multi-modal representation (which can be neurons, but is implied to be substrate-independent according to Neurorepresentationalism). One way to cash out this idea as an ontology is as some type of neutral monism: There is some reality out there, which we cannot access directly, and there are different ways to interpret or ‘decode’ it. On the one hand, one could decode this reality at the physical level, namely, as fundamental particles, atoms, neurons, etc. Another possibility would be to decode it as conscious representations. The concept of a neuron might not have to be evoked in the latter case. Please note that this ‘neutral monism interpretation’ is my own interpretation and is not proposed by Pennartz.

Exclusivity

Earlier I mentioned Neurorepresentationalism’s claim that, for a multi-level representation to be conscious, it needs to encompass more than one modality. This implies that the lowest representational levels cannot be conscious (only as part of higher levels).

However, it does not imply that we need to reach the highest level before consciousness arises. This is because multi-modal meta-networks can arise at intermediate stages in the representational hierarchy. Still, Pennartz proposes that consciousness arises solely on the highest level, which suggests that he is assuming a type of ‘exclusivity’ rule: Individual sub-components of a conscious representation cannot also be conscious on their own.

This shows a lot of similarities with Integrated Information Theory’s axiom of exclusivity [4]. To me this constitutes an extra assumption that we don’t necessarily have to make. A theory that leaves open the possibility of ‘sub-experiences’ having an existence of their own might seem strange, but it carries fewer assumptions.

Representing Everything with Anything

In the previous section, I gave a short (and limited) account of Neurorepresentationalism, combined with some ideas about how its claims could be cashed out as an ontology about physics and consciousness. Next, I want to zoom out a bit and explore some potential implications of strong representational theories of consciousness [2]. While spooky and speculative, I believe they deserve serious consideration.

Assume we are looking at a conscious representation of a visual percept of an apple. This could be a set of neurons with a particular firing pattern, or a state (or sequence of states) in a digital computer. While the possibility for consciousness on a digital computer is contested, a strong representational theory that makes no claims about the substrate of representations may allow for this (as does Neurorepresentationalism). Let us also assume our representational theory of consciousness is narrow, meaning that the conscious content is only dependent on the representation, rather than dependent on what’s going on outside of the agent [2]. In other words, two agents with the same internal representations will have the same experience. This means that our apple percept is fully determined and instantiated by the apple representation. Looking at our neurons or transistors, how can we tell that the representational content corresponds to an apple instead of an orange? Or, more broadly, how can we tell that it corresponds to the modality of vision at all? Clearly, you might say, given the right decoding scheme, we can easily extract that information. However, who decides the decoding scheme? How does the subjective experience of the apple ‘know’ how it needs to decode its underlying representation? Minus the consciousness, this is equivalent to saying that a random string of bits can be decoded to any arbitrary file, given the right file format. Why, then, does my conscious experience ‘know’ it’s experiencing any given experience instead of an arbitrary other one? Pardon all the questions, but here is the final one: If anything can represent everything (given the right decoding scheme), how could some system ever have a definite conscious experience? There are a lot more nuances to this line of reasoning, and I highly suggest reading Greg Egan’s Permutation City [8] (a hard sci-fi novel packed with a huge number of crazy ideas, culminating in one very trippy theory) and/or Hans Moravec’s essay Simulation, Consciousness, Existence [9].

I know, there are a few steps made here that rely on a very simple notion of the word ‘representation’. However, if the above reasoning is wrong, I think it’s important to be explicit about why it fails, which would reveal further assumptions we place on what types of representations can be conscious.

How might Neurorepresentationalism avoid this hyper panpsychist universe where anything can encode any conscious experience? Well, you might say, Neurorepresentationalism puts a lot of constraints on what kinds of representations are conscious: They have to be multi-modal, multi-level, etc. But remember, being substrate-independent, Neurorepresentationalism could be implemented on anything, including computer chips. Moreover, whether a string of bits encodes a multi-modal representation or a high-resolution photo of yourself is entirely dependent on the encoding scheme. Thus, unless we put further assumptions on how these representations are instantiated in the physical world, or how they are encoded, we cannot avoid finding multi-modal representations wherever we want! There may be very intuitive assumptions that we could make to avoid this scenario, such as requiring some continuity across time. For example, we might require the probability distribution of the visual part of the multi-modal representation to follow the statistics of visual scenes. We might also require different parts of the representation to have some causal effect on each other (similar to IIT [4]). In both cases, we would move away from a static conception of representations. It’s quite likely that similar assumptions are being made with Neurorepresentationalism, but I think it’s worth thinking about this explicitly.

Conclusion

Representationalism provides a reasonable perspective from which to approach a scientific theory of consciousness. After all, few people doubt that our perception of an apple is deeply connected to some internal model we have of that apple.

Neurorepresentationalism proposes a theory for how such representations are structured, and how they relate to subjective experience. Making clear claims about neural correlates of consciousness, a theory like this lends itself to scientific study according to the framework of the real problem. Moreover, by treating subjective experience and physics as equally real, but corresponding to each other through a hierarchy of representations, we are avoiding any strong assumptions about the ontological primacy of one over the other.

However, as with all strong representational theories, we have to be careful about our requirements for conscious representations. Barring any further constraints on continuity or causality, representational views of awareness quickly lead us to hyper-panpsychist scenarios which, while not necessarily impossible, seem quite extreme.

Bibliography

[1] Pennartz, Cyriel MA. “What is neurorepresentationalism? From neural activity and predictive processing to multi-level representations and consciousness.” Behavioural Brain Research 432 (2022): 113969.

[2] Lycan, William, “Representational Theories of Consciousness”, The Stanford Encyclopedia of Philosophy (Fall 2019 Edition), Edward N. Zalta (ed.), URL = https://plato.stanford.edu/archives/fall2019/entries/consciousness-representational

[3] Chalmers, David J. “Facing up to the problem of consciousness.” Journal of consciousness studies 2.3 (1995): 200–219.

[4] Oizumi, Masafumi, Larissa Albantakis, and Giulio Tononi. “From the phenomenology to the mechanisms of consciousness: integrated information theory 3.0.” PLoS computational biology 10.5 (2014): e1003588.

[5] Koch, Christoph. “What is Consciousness?” nature.com URL = https://doi.org/10.1038/d41586-018-05097-x

[6] Chis-Ciure, Robert, and Francesco Ellia. “Facing up to the hard problem of consciousness as an integrated information theorist.” Foundations of Science (2021): 1–17.

[7] Frankish, Keith. “Illusionism as a theory of consciousness.” Journal of Consciousness Studies 23.11–12 (2016): 11–39.

[8] Egan, Greg. “Permutation City”. Millennium Orion Publishing Group, 1994.

[9] Moravec, Hans. “Simulation, Consciousness, Existence” (1998) URL = https://frc.ri.cmu.edu/~hpm/project.archive/general.articles/1998/SimConEx.98.html

Originally published in Better Programming on Medium.