Your Brain Does Not Think in Words

A man called G.S. in the research had a stroke on the left side of his brain at 48. He has a graduate degree and he used to work in cybersecurity. Afterward he could not understand spoken sentences. On a test of 80 sentences where random guessing gets you 40 right, he scored 22. On the subset built out of passive sentences, the "the dolphin was splashed by the diver" kind, he got 2 out of 40.

Then they gave him logic puzzles. Look at a list of numbers, look at what it turns into, work out the hidden rule. He solved 39 of the 40 rules they put in front of him. On a standard deductive reasoning test he scored 26 out of 30, which is 2.3 standard deviations above the average for his age. A second patient, S.A., 78 years old and 33 years past a brain infection that took his language, solved 19 of his 25 rules and scored 25 out of 30 on the same deduction test, 1.8 standard deviations above average for his age.

Read that again. Both men lost the ability to parse a sentence and then beat the average person their age at formal logic. Neither had the words to explain the rules he had found, so they pointed and drew pictures.

The study is Evidence from formal logical reasoning reveals that the language of thought is not natural language, published in PNAS on July 6 by Hope Kean, Evelina Fedorenko, and colleagues at MIT with Rosemary Varley at UCL. There is a free PDF on bioRxiv if you hit the paywall, and MIT's own summary is the fastest read. I found it through LaurieWired's post, which passed 100,000 views in an afternoon, and her framing is why I am writing this.

The brain scans say the same thing

The team also put 29 healthy people in an MRI machine and gave them the same families of problems. Find the hidden rule, which is the puzzle you solve by looking at a few examples and working out what they have in common. And check whether a conclusion follows, the "if A is true, then is B true" kind.

The language areas of the brain never turned on. Then the researchers cranked the difficulty up. If words were doing the reasoning, harder logic would light those areas up more. They stayed flat.

The brain also runs a general problem-solving system for hard unfamiliar tasks. It showed up for the find-the-rule puzzles and skipped the if-then problems, which surprised the researchers and tells you the logic is running somewhere the field has not fully mapped.

Your logic runs on its own, in its own format, and language is the delivery service. Words carry the problem in and carry the answer back out. Words do not do the thinking. Kean's line about the old theory that you need language to figure out rules: "This really upends" it. She gets to say that because nobody can claim the language system was secretly helping when the language system is destroyed.

Why this is a problem for chatbots

Here is the argument LaurieWired made, and it is the clearest version of the case against "just make it bigger" that anyone has aimed at a normal audience this year.

Every model you use was trained on one thing: text. Words in order, billions of them. That is the only raw material any of them has ever had. Human logic does not run on words, so a machine that only has words is missing the equipment we reason with. When a model looks like it is reasoning, what is happening is that it is finishing a pattern in text, extremely well.

Text is a sloppy medium for exact work. It evolved to move an idea out of my head and into yours fast, not to keep every logical step airtight. The same word means different things in different rooms. Half of any sentence is implied and never said.

That explains something you have already lived through at work. The model sounds completely on top of a problem, you change three words in the prompt, and the answer flips. It aces the version of the problem it has seen a thousand times and falls over when the same problem arrives with different variable names. It writes an argument that reads beautifully with one step in the middle that is flatly wrong. If real reasoning were happening underneath, the wording would be a paint job. Change the words and you change the answer, because the words are the machine.

The counterargument, and why I do not buy it

This study looked at human brains. It did not test a single model, and I am not going to pretend it did. My claim is about where to put your money and your architecture, not a proof about what silicon can do.

So take the best objection: a machine does not have to copy the human method to get the human result. Planes do not flap their wings. Different guts, same job, and that is normal in engineering.

Fine. Now name the pure-text system that holds up on formal work. There isn't one. Every AI system that survives contact with production hands the exact part to something exact, and it has been that way for three years running while the models got twenty times bigger. Planes do not flap, and they still needed a wing. Text is not the wing.

The useful version of all this

Look at what the reliable systems actually do. The model writes code and a real interpreter runs it. The model calls a calculator, a database, a type checker, a test suite. The precision lives in the tool. The model does translation: your words in, structured problem out, plain answer back.

People treat that as a temporary patch while we wait for models big enough to skip it. It is not a patch. It is the shape of the thing, and this study is why. So the move, whether you are picking an architecture or defending one in a design review, is to shove every step that has to be correct into a system that can be checked, and let the model do the part words are good at.

When a vendor tells you the next model will not need any of that, ask where the reliability comes from. If the answer is the model itself, they are betting your on-call schedule on the one part of the stack this study just knocked out from under them.

Who pays when the bet is wrong

This is not a seminar-room debate, because these systems already sit in front of decisions about real people. Insurance claims get denied. Resumes get filtered out. Incidents get triaged. Headcount plans get approved on a confident-sounding summary nobody upstream can check.

A machine that is fluent and quietly wrong is the worst possible fit for that work, and it is going in anyway, because fluent is what the buyer sees in the demo.

When it fails, it does not fail for the executive who signed the contract. It fails for the person whose claim got denied and who now has to fight a support queue to fix it. It fails for the engineer paged at 3 a.m. behind an agent's confident bad call. It fails for the coworker whose job was cut on a projection that sounded rigorous. That is the game. The upside goes up and the error rate comes down on people with no way to appeal it.

A man who cannot understand the sentence "the diver splashed the dolphin" drew the answer on a piece of paper and beat the average person his age at logic. Whatever he was using, we do not have it in a datacenter. Build like that is still true.

Get good.

- Dr. J