Your Brain Does Not Think in Words

Two people had strokes that wrecked the language parts of their brains. Afterward they could not reliably understand a sentence as simple as "the teacher finds the child." One of them got about a third of those grammar questions right, which on a two-choice test is worse than flipping a coin.

Then researchers gave them logic puzzles. Find the hidden rule that turns one list of numbers into another. Finish the pattern in a grid of shapes. Both of them scored about as well as healthy adults. They had no words left to explain the rules they figured out, so they pointed and drew pictures instead.

That study is Evidence from formal logical reasoning reveals that the language of thought is not natural language, published in PNAS on July 6 by Hope Kean, Evelina Fedorenko, and colleagues at MIT, working with Rosemary Varley at UCL. There is a free PDF on bioRxiv if you hit the paywall, and MIT's own summary is the fastest read. I found it through LaurieWired's post yesterday, which passed 100,000 views in a few hours, and her take is why I am writing this.

The brain scans are the part that should stop you

Two patients is two patients. The other half of the study is harder to shrug off.

The team put healthy people in an MRI machine and gave them the same kinds of problems. Two kinds, specifically. Find the hidden rule, which is the puzzle you solve when you look at a few examples and work out what they have in common. And check whether a conclusion follows, the "if A is true, then is B true" kind of problem.

The language areas of the brain never turned on for either one. Here is the detail that matters most. The researchers made the puzzles harder and harder, and the language areas still did nothing. If words were doing the reasoning, harder logic should light them up more. It did not light them up at all.

The brain also has a general problem-solving system, the one that handles hard unfamiliar tasks. It showed up for the find-the-rule puzzles and then, to the researchers' surprise, skipped the if-then ones too.

So the picture looks like this: your logic runs somewhere on its own, in its own format, and language is the delivery service. Words carry the problem in and carry the answer back out. Words are not what does the thinking. Kean's line about the old theory that you need language to figure out rules: "This really upends" it. She can say that because you cannot claim the language system was secretly helping when the language system is gone.

Why this is a problem for chatbots

Here is the argument LaurieWired made, and it is the clearest version of the case against "just make it bigger" that I have seen aimed at normal people.

Every model you use was trained on one thing: text. Words in order, billions of them. That is the only raw material it has ever had. If human logic does not run on words, then a machine that only has words is missing the equipment we actually reason with. When a model looks like it is reasoning, the honest description is that it is finishing a pattern in text very, very well.

And text is a sloppy medium for exact work. It evolved to get an idea out of my head and into yours quickly, not to keep every logical step airtight. Same word means different things in different rooms. Half a sentence is implied and never said.

That would explain something you have already lived through at work. The model sounds completely on top of a problem, then you change three words in the prompt and the answer flips. It aces the version of the problem it has seen a thousand times and falls over when the same problem shows up with different variable names. It writes an argument that reads beautifully and has one step in the middle that is just wrong. If real reasoning were happening underneath, the wording would be a paint job. The wording is not a paint job. Change the words and you change the answer, which is what you would expect if the words were the whole machine.

Where I have to be honest with you

Flag on the play, and it is my own side's flag. This study looked at human brains. It did not test a single AI model, and it does not prove that AI cannot reason. Two patients is a small group, and the authors are careful about that in a way the internet will not be this week. What they showed is that these kinds of logic survive without language, which is a big deal, and not that every kind of thinking works this way.

There is also a fair comeback. Nobody says a machine has to copy the human method to get the human result. Planes do not flap their wings. Different guts, same job, happens all the time in engineering. "Brains do it another way" is a fact about brains.

What the study gives us is a design idea worth taking seriously: work that has to be exactly right may need its own kind of representation, separate from the words we use to talk about it. And that idea matches what already works in the real world.

The useful version of all this

Look at the AI systems that actually hold up in production. Every one of them hands the exact part to something exact. The model writes code and a real interpreter runs it. The model calls a calculator, a database, a type checker, a test suite. The precise work happens in the tool. The model does translation: your words in, structured problem out, plain answer back.

People treat that setup like a temporary patch while we wait for the models to get big enough to skip it. This study suggests it is not a patch. It is the shape of the thing. So the move, whether you are picking an architecture or defending one in a design review, is to shove every step that must be correct into a system that can be checked, and let the model do the part words are good at.

And when a vendor tells you the next model will not need any of that, ask where the reliability is coming from. If the answer is "the model," they are betting your on-call schedule on the one thing this study just made shakier.

Who pays when the bet is wrong

This is not a debate for a seminar room, because these systems are already sitting in front of decisions about real people. Insurance claims get denied. Resumes get filtered out. Incidents get triaged. Headcount plans get approved on the strength of a confident-sounding summary nobody upstream can check.

A machine that is fluent and quietly wrong is the worst possible fit for that work. It is going in anyway, because fluent is what the buyer sees in the demo.

And when it fails, it does not fail for the executive who signed the contract. It fails for the person whose claim got denied and who now has to argue with a support queue to fix it. It fails for the engineer paged at 3 a.m. behind an agent's confident bad call. It fails for the coworker whose job was cut based on a projection that sounded rigorous. That is the game. The upside goes up and the error rate comes down on people who have no way to appeal it.

Two people with no words left drew their answers on a piece of paper, and they were right. Whatever they were using, we do not have it in a datacenter yet. Build like that is still true.

Get good.

- Dr. J