Large Language Models 6: Philosophical Investigations
This is the sixth post in a series on large language models. The first one is here.
We have to talk about the Chinese Room.
Searle was right.
LLMs really are Searle’s Chinese Room built. Whether that meets your definition of brain or mind is kind of up to the user of language (now you can get into Wittgenstein!). But currently the definitions of brain and mind and intelligence are used in different contexts with different meanings it’s quite hard to parse exactly what someone is thinking when they say any of these words.
The issue with the Chinese Room metaphor is that you still intuitively have a person like thing doing the lookup - which is likely to confuse people beacuase of intelligence and intentionality leak in.
Plenty people might want to argue about the “brain-ness” or “mind-ness” of the room. You can then get into arguments about embodiment, extended mind and so on. It’s not really explanatory for the specific technology of LLMs, but the philosophy works the same.
Either way, the Chinese Room has literally been built. The books in the room were replaced with a fancy vector database of all linguistic connections and the mindless robot inside with a complicated the transformer architecture using attention mechanisms and feed-forward networks. But it still just gets words, looks stuff up in the database and sends words back.
Searle was right. There is a symbol grounding problem here.
Searle was wrong
The Chinese Room fails as a description of mind because it fails to work at a level of description that maps correctly. Minds are not isolated but a vast shared tool created over thousands of years by embedded embodied animals. The Chinese Room of language processing is a tool that helps a mind think, just as a sculptor uses a chisel to sculpt. It is not the thinking itself.
The sculptor uses a chisel shaped for a human hand that leverages the muscles of the human body to change the shape of another object. The Chinese Room is nothing without the wetware of a human using it - and the room itself reaches into the wetware and shapes it. This is similar to the way a violinist gets hard skin on their fingers that help them play the violin. We are already merged with our technology - and language is the most fundamental and oldest of these.
Really, the “magic” is the reconstruction from lossy data is necessarily generative and thus seems “intelligent” because it returns novel text. Now I’d claim it’s not magic or even mysterious at all but demonstrates the extended mind thesis.
If it works it’s computer science. If it doesn’t work it’s A.I.
In some ways this entire series of posts is an updated version of the oldest joke in AI reformulated:
If it works it’s computer science. If you don’t know how it works it’s A.I.
Returning to where we started: Stephen Wolfram noted the big surprise that LLM’s provide is that they seem to demonstrate that language is fundamentally simpler and more law like than previously suspected.
Andy Clark and David Chalmers are likely to be less surprised; their extended mind thesis addresses this.
One of my favourite lessons at University was a robotics demonstration. We were shown a Roomba like robot with complicated wires and sensors. It was hacked together with, a few bits connected to a circuit board that looked like a Raspberry Pi. We were asked, based on its behavior how many lines of code does the robot need to find the door and exit the room.
It was placed in the center of the room, and we watched it set off just like a robot vacuum cleaner, bouncing around between chair legs and feet until it went out the door.
There were many guesses at the code base size - 5 lines to 500.
Then the professor then removed the lid. The robot wasn’t wired. It wasn’t even electrical, it was wind up.
It contained nothing but wheels with differential friction that, when driving unobstructed goes straight forward. But hit a wall or object and one wheel spins harder, and that extra power changes the direction of the robot, driving away from the wall at a random angle. Eventually it gets lucky, and drives out the door.
What appears as problem-solving intelligence is merely physical properties interacting with the environment—no code, no planning, no representation.
The extended mind thesis argues that cognitive processes aren't confined to the brain but extend into our environment through external scaffolding, with language being perhaps the most powerful cognitive technology we've developed. Clark and Chalmers show how we offload cognitive work onto environmental structures, creating hybrid brain-world systems.
LLMs exemplify this principle from a different angle. Their power doesn't stem from mysterious self-aware alien minds but from capturing the patterns of how we embed thinking in language itself. Language functions like an abacus for thought—a technology for manipulating and structuring ideas.
This doesn't diminish what LLMs can do, but does demonstrate why they can fail spectacularly at tasks that seem trivial. They operate on the patterns embedded in language without access to the grounded, embodied experiences that shape human cognition. We offload ideas and thoughts to language structures to help us think.
Language itself is a game played between minds (Wittgenstein’s insights are very relevant to the structure of language models). Meaning emerges from language used in social contexts, not from fixed definitions. Human language presupposes the use of words in shared embodied experiences - that is where meaning in generated. There's a fundamental gap between statistical patterns in text and the situated understanding and intentionality that comes from interacting with the world using a body.
Minds are the product of the dynamic interaction between brains and environment—intelligence distributed across grey matter, social structures, tools and technologies. What might look like intelligent processing in both humans and machines often emerges from simpler components interacting with structured environments..
What might look like intelligent processing can just be friction and some wheels.