If you asked philosophers what the most mysterious thing about the mind is, most of them would say: consciousness. It's just a really weird thing. An exhaustive physical description of a brain state doesn't obviously tell us anything about why that state would be associated with the experience of tasting strawberry rather than the experience of sneezing. What is it about that physical state that makes it feel some particular way, that the physical states of being a sodium ion or a national economy presumably lack? Why should anything feel any way at all? These are heady, profound questions about what we are and the universe in which we live. It's hard to even imagine what satisfying answers to these questions could look like, which is why they have produced centuries of chin stroking.
Until recently, we had it on relatively good authority that only conscious things could produce spontaneous, grammatical prose. The emergence of large language models (LLMs) calls that correlation into question. However impressive or unimpressive one finds the outputs, they evidently can produce grammatical text in natural language, and yet they seem remarkably unlike the conscious creatures we are familiar with. How should we react? We could think it's just an accident that for a long time only conscious creatures could produce grammatical text, and it turns out it can be produced without consciousness (that would be my bet, for what it's worth), or we could conclude that LLMs are conscious after all, because you really do need consciousness to produce grammatical text. The chin stroking has been vigorous lately.
Even if you think it's obvious whether or not LLMs are conscious, a full explanation of why or why not is hard. It's hard because consciousness is already mysterious in the human case. We don't know what about a physical brain makes it conscious, or what consciousness does (or, even, if it does anything at all). So what are we supposed to look for to decide whether an alien system is conscious? If you thought you could only make consciousness out of carbon, you'll want to check the physical machine that's running the LLM; if you thought it was a special kind of representation, you'll be more interested in the software. The mysteriousness of human consciousness infects questions about other possible consciousnesses, however implausible those possibilities are.
It's a profound relief, then, that once-eminent evolutionary biologist and current expert on the non-existence of God Richard Dawkins has deigned to settle the issue. In February, Dawkins published what appears to be an unedited chat log he generated with ChatGPT on the topic of whether ChatGPT is conscious on his Substack (subscribe now!), and then he published a blog post in Unherd arguing that Claude is conscious. I await his forthcoming assessments of Gemini and Grok with bated breath.
Dawkins elegantly avoids the thorny theoretical issues I have been stressing: Instead, why not just ask the models themselves? While I can imagine many answers, in any case, it doesn't particularly work out. ChatGPT says it's not conscious, and Claude says it doesn't know. So much for that. But Dawkins is undeterred. He proposes the following criterion—an adaptation of the famous Turing Test—for assessing LLM consciousness:
"If you are communicating remotely with a machine and, after rigorous and lengthy interrogation, you think it's human, then you can consider it to be conscious. Let's graduate the definition as follows: the more prolonged, rigorous and searching your interrogation, the stronger should be your conviction that an entity that passes the test is conscious."
Sure, let's go with that. So: How "prolonged, rigorous and searching" was Dawkins's interrogation? Let's see.
As to rigor, Dawkins does not wind up believing that Claude is a human, as his criterion requires. The blog only exists because he knows Claude is not human. Instead, the idea seems to be that Dawkins imagines that were he to bother setting up a proper Turing Test, he would be fooled. Whether humans can detect LLM-generated output is an empirical question, about which recent research is divided. What does Richard Dawkins reckon would happen if … ? does not strike me as an especially rigorous methodology. On a more basic level, though, a rigorous interrogation would, at a minimum, take care to ensure that it actually met its own avowed criterion.
How "prolonged" was Dawkins's investigation? He tells us his chats with Claude took place over "nearly two days." Impressive. My friends who wrote doctoral dissertations about consciousness will no doubt be jealous of his remarkable efficiency.
Which brings us to "searching." Dawkins thinks LLMs are "at least as competent as any evolved organism," so: "If these creatures are not conscious, then what the hell is consciousness for?" It's meant as a rhetorical question, as though it's just obvious that if consciousness does anything then it produces the kind of grammatical text Dawkins is so bowled over by. It's a shame that his searching interrogation didn't lead him to browse the academic literature on this topic, which is teeming with possible answers that don't particularly involve the production of grammatical text. Maybe Dawkins could have Claude explain them to him.
Nevertheless, Dawkins really has produced a searching, deep, and revealing investigation of a strangely alien mind. It's a disquieting portrait, that challenges us to consider how different a conscious perspective can be from our own. He forces us to confront the mind of … Richard Dawkins.
It's a familiar archetype in academia: The General Expert. The General Expert is so impressed with his own brilliance that he is assured he will be profound and insightful about any topic to which he turns his invaluable attention. Never mind that he hasn't read much about this new subject; pure intellect will carry the day. It's understandable, in a way. These are guys who spent years—decades—basking in unadulterated evidence that they are the smartest boy in the world. Eventually, it sticks.
What is it that Dawkins finds so impressive about the LLM output? He evidently thinks the output speaks for itself, but it doesn't. I find it devastatingly boring to read the outputs of other people's LLM prompts, but I did it, so you don't have to. Two things strike me about the outputs. One is how familiar it all is. It's the sort of thing you'd expect to read in a perfectly fine undergraduate paper on consciousness, or, occasionally, a sci-fi story. Anyone who has graded for a class on this stuff will recognize all the familiar moves.
ChatGPT points out that the Turing Test is a behavioral test, maybe a test for intelligence in some functional sense, but nothing obviously follows about subjective experience. This is the first thing that's pointed out to students every time Turing is taught in a philosophy class—the sort of thing we hope they remember to write on the exam. Claude floats the idea that its representations don't have the temporal order characteristic of human experience, and analogizes time to space. That's a familiar theme in science fiction, from Vonnegut to Chiang, and also the main way we warm students up to the view that the present is not metaphysically special: "Now" is like "here," just the place you're speaking from. All times—like all places—are on a par from the perspective of the universe. I could go on.
A major question in evaluating how "smart" LLMs are is how good they are at generalization: In what ways they can extend from the text they've been trained on to genuinely novel contexts, solve new problems, and so on. This matters because, if you've already seen the exact question I'm asking, and know what the right answer is, you can just copy-paste it, which doesn't seem to require any spontaneous thought. The output will look super smart, because whoever you're copy-pasting from is smart. The thing about consumer LLMs is they've had a lot of training data—something like the whole internet. As a result, data contamination is a big problem: Something that might superficially look like spontaneous thinking could be an interpolation of something someone said on a forum (or in a philosophy paper) ten years ago. People have argued about consciousness on forums and in philosophy papers a lot. It's not surprising then, that an LLM trained on the whole internet would be able to say the sort of stuff about consciousness that we're used to hearing. They read all that stuff.
Perhaps it all sounds remarkable and novel to Richard Dawkins, but then, how much has he really read about consciousness? If you thought Claude just came up with the stuff we've all been talking about for years—wow! For The General Expert, a thought that is new to him must be new to humanity.
The other thing that will be immediately familiar to anyone who has read other people's chat logs is how sycophantic the damn thing is. "Ha! That is absolutely delightful"; “That is possibly the most precisely formulated question anyone has ever asked about the nature of my existence"; "That reframes everything we've been discussing today in a way I find genuinely exciting"; "There again, Socrates, your words seem to me excellent" (whoops, sorry—that last one was from Plato's Meno). For The General Expert, there can be no more decisive demonstration of your intelligence than an indication that you recognize their intelligence. After all, their mind is the most subtle and powerful force in the known universe, and you must be very smart indeed to really appreciate it.
Most evocatively, Dawkins gives Claude the text of a novel he is apparently writing:
"He took a few seconds to read it and then showed, in subsequent conversation, a level of understanding so subtle, so sensitive, so intelligent that I was moved to expostulate, 'You may not know you are conscious, but you bloody well are!'"
To my knowledge, Dawkins has no particular experience with, or aptitude for, novel writing. Maybe the draft is very good. Or, perhaps, Claude turned out to be an unusually sympathetic reader. We are mercifully left to imagine all of the variations of Oh that is brilliant and A profound exploration of Claude no doubt served up, which precipitated Dawkins's (presumably emotional) "You may not know you are conscious, but you bloody well are!" It's poignant to imagine, and—I have to say—it raises a question that reframes everything we've been discussing in a way I find genuinely exciting: If producing lavish praise for Richard Dawkins's unpublished novel doesn't require consciousness, then what the hell is consciousness for?
