The gap between intelligence and understanding

Your phone lights up. An unknown number, and beneath it two small words: Possible spam.

It is tempting to trust the warning. The filter is clever, and the label sounds sure of itself. Before you read on, make a guess.

If you guessed far above 8.3%, you are in good company. When the psychologists Gerd Gigerenzer and Ulrich Hoffrage gave problems like this to university students who had never met Bayes’ theorem, only about one answer in six followed the right reasoning. Earlier studies, they noted, had found physicians struggling with the same kind of problem.^[1]

Now look again at the answer itself. It is correct. It is complete. It even shows its working.

And yet, unless you already think in conditional probabilities, it probably did not make the result feel obvious. You could repeat it. Explaining it to a colleague is another matter. So is saying what would happen if spam calls became more common.

This is the feeling I have most often when I read long answers from AI models. Each sentence is fine. The trouble lives between the sentences. I am left to work out what matters most, how the pieces connect and what the whole thing looks like.

Set the probabilities aside for a moment. Picture 1,000 ordinary calls arriving at the phone, and watch which ones it flags.

Of the 108 warnings, only 9 are for spam.

The reason was hiding in plain sight. Other calls vastly outnumber spam, so even a small error rate on them produces most of the alarms. Once you can see that crowd of ordinary calls, the answer stops being surprising. It becomes almost obvious.

Nothing was added. These are the numbers the formula used. Only the shape changed: counts of things instead of probabilities of events, and a picture in which the large group stays visibly large.

Psychologists have measured how much this matters. In Gigerenzer and Hoffrage’s first study, changing only the format, from “1%” to “10 out of 1,000” and so on, lifted the share of correct Bayesian reasoning from 16% to 46% across fifteen different problems. Nearly three times as many.^[1] A later review of 35 articles confirmed the advantage and found that visual aids help too. It also found a sobering limit: even with frequencies, most people still get these problems wrong.^[2]

I think this small example captures something large about AI. A model can produce the correct answer. Whether a person can use it depends on something else: how the answer meets that person’s mind. Model capability and human understanding are different outcomes. When understanding matters, we should build for both, and measure both.

Not every question needs a lesson, of course. If you want a translation, a train time or a line of code that works, an explanation can get in the way. But when the purpose is to learn, to check a recommendation or to make a decision, an answer you cannot use is not yet finished.

Nor do impressive test scores settle the matter. A high score on a hard exam tells us something about the model. It does not tell us whether a particular reader walked away able to use what it wrote.

My suspicion is that the gap widens as models improve. They can now answer questions whose answers are harder to absorb, and those answers still arrive as blocks of text. I hold this as a hypothesis, not a measured trend.

The picture worked where the formula did not. Part of the reason is how little we can hold in mind at one time.

Psychologists call this workspace working memory. Think of it as a small mental desk, where we hold and combine whatever we are thinking about right now. It has room for only a few new items.

Knowledge makes the desk feel bigger, because familiar patterns take up less room. A statistician sees “0.9 × 0.01” as one familiar move. A newcomer sees several separate pieces, and must keep them all in view while working out how they relate. John Sweller’s research on cognitive load tied these limits to learning: the effort spent just getting through a task can crowd out the effort that builds lasting knowledge.^[3]

Every unfamiliar term in an answer opens a thread that the reader must hold until the text ties it to something familiar. Sensitivity. Specificity. Positive predictive value. Open several threads before connecting any of them, and you ask the reader to build the structure and understand it at the same time.

Research on learning from words and pictures offers practical remedies. Mayer and Moreno describe several: present a lesson in segments the learner can pace, introduce the key parts before the relationships between them, and place words beside the part of the picture they describe.^[4] Those results come from carefully built lessons, not chat answers. Carrying them over to AI is a sensible starting point, and one that still needs testing.

Then comes a twist. Help that rescues a beginner can get in an expert’s way. Kalyuga and colleagues call this the expertise reversal effect.^[5] The statistician does not need 1,000 squares; the formula is faster. So there is no single best explanation, only a good explanation for a particular reader and purpose.

And expertise is local. The same statistician may need the beginner’s route the moment the topic turns to contract law.

If the only goal were to make answers feel easy, AI would already be close to perfect. Its answers are fluent, confident and neatly formatted. The trouble is that the feeling of understanding is a poor guide to the real thing.

You can test this on yourself in about a minute.

Leonid Rozenblit and Frank Keil ran this experiment with Yale graduate students, using everyday devices such as zippers, flush toilets and cylinder locks. Participants rated their understanding, wrote out step by step how each device works, then rated it again. Nearly everyone’s ratings fell. Many said they were surprised by how little they knew.^[6]

The researchers called this the illusion of explanatory depth. It is strongest for explanations of how things work, and much weaker for facts, procedures and stories.

Polish can create a similar illusion. In one study, students watched the same one-minute explanation of why calico cats are usually female. Half saw the speaker deliver it smoothly: upright, making eye contact, without notes. The other half saw the same speaker hunched over notes, hesitating. The smooth group expected to remember far more, and rated the speaker as better prepared and more effective.

On the memory test, the two groups did about the same.^[7]

The reverse happens too. At Harvard, physics students were randomly split between polished lectures by highly rated instructors and active-learning classes on identical material, and then swapped. In the active classes they scored almost half a standard deviation higher on the test, yet reported feeling that they had learned less.^[8] The researchers traced part of the mismatch to effort. To the students, harder thinking felt like learning less.

AI summaries show a similar pattern. In seven experiments with more than 10,000 people, some learned about a topic, such as how to plant a vegetable garden, from an AI-style summary, and others from ordinary web links. The summary readers spent less time with the material, said they had learned less, and wrote sparser, more generic advice for a friend. In one experiment both groups received exactly the same information; only its format differed. The pattern held.^[9]

These measures were self-reports and the advice people wrote, not a knowledge test. Still, the direction is consistent: what is easy to receive can be thin to keep.

Two findings stop this from becoming a case for friction. When Buçinca and colleagues made people think before seeing an AI’s suggestion, people relied less on its wrong advice. But they liked those designs least, the benefit varied between people, and overall performance did not significantly improve.^[10] And in nutrition experiments by Gajos and Mamykina, people given only an explanation learned from it, while people given a recommendation together with the same explanation did not show the same learning.^[11]

What matters, then, is less the amount of effort than what the person is left to work out.

What happens when a modern model walks into a classroom? Two recent studies give a vivid answer, and it depends on design.

In a large high school in Turkey, Hamsa Bastani and colleagues randomly assigned about fifty maths classes, nearly 1,000 students in all, to practise in one of three ways. Some classes had no AI. Some had a plain GPT-4 chat assistant. Some had a GPT-4 tutor instructed to give teacher-designed hints instead of answers. Each session ended with a closed-book exam on the same ideas.^[12]

Use the button in the figure to take the AI away.

With the AI at hand, both groups beat the no-AI group in practice: grades were 48% higher with the plain assistant and 127% higher with the tutor.

Then the AI was gone. Students who had practised with the plain assistant scored 17% lower than students who never had AI. The tutor group’s exam scores could not be distinguished from those of the no-AI group.

Two details make this study hard to forget. Many students used the plain assistant as a crutch, asking for solutions and copying them, even though its answers were correct only about half the time. And students misjudged their own learning. Those who used the plain assistant did not think they had learned less. Those who used the tutor believed they had done better on the exam. They had not.

Design can also tip the result the other way. Some of the same Harvard researchers behind the active-learning study built an AI tutor around research on how people learn, with pre-written, step-by-step solutions behind it. In an introductory physics course of 194 students, they compared it with an active-learning class. On the test straight after the lesson, the tutored students scored higher: a median of 4.5 against 3.5. They usually needed less time, too: a median of 49 minutes against the class’s 60.^[13] The study covered two lessons and tested learning immediately, so it cannot say how long the gains last.

Read side by side, the two studies support a narrower and more useful conclusion than “AI makes us learn less” or “AI teaches better”. Learning depends on the whole arrangement: the model, the information it has, how it is told to teach, the interface, the task and the learner. Neither study isolated wording or layout, so neither shows that dense prose is the culprit.

If understanding is part of the goal, an answer should give the reader a route through the subject, not just a destination. Step through five changes below and watch what each one does.

1. Lead with the point.

   Say what matters most and why, in a sentence the reader can keep. Then show how the supporting parts fit together. A heading cannot supply a missing connection.

2. Open depth on demand.

   Let the reader open the calculation, the assumptions or a source without losing their place. Keep any assumption that changes the meaning in plain view.

3. Fit the purpose.

   Someone deciding how to respond to a flagged call needs a different route from someone learning the idea or checking the arithmetic. Let readers choose, and let them change their minds. Asking for a short answer does not make someone an expert.

4. Show the relationship.

   A picture can show what a paragraph only asserts, and a slider can expose an assumption. Decoration, needless motion and crowded dashboards do the opposite: they add work.

5. Check, gently.

   Asking for a single prediction can reveal a specific misunderstanding, so the answer can respond to it. But one right answer, a pause or a model’s confidence is not proof that someone understands.

W3C’s guidance on cognitive accessibility points the same way, recommending manageable amounts of content and clear choices.^[14] That is a sound basis for design, though not a guarantee that a particular interface teaches well.

None of this is an argument against text. Text is precise, searchable, easy to revisit and often the most accessible format there is. The opportunity is to add other shapes when they help. A more capable model may itself become better at choosing them, so model progress and interface design can improve together.

There is a risk, too. A beautifully structured wrong answer is more persuasive than a messy one. So a better presentation needs a route to verification: sources and assumptions the reader can inspect, not decoration that makes the answer look authoritative.

How would we know whether any of this works? Not from how much readers like an answer, how long they stay or how often they press “I understand”. Those measure the interaction. As the studies above show, they can even move in the opposite direction from learning.

We need tests of what people can do. Here are four, using the phone example.

A real study would go further. It would start with a specific task and verified information, then randomly assign readers to an excellent static explanation, a conventional chat answer or an interactive version, taking account of what each reader already knows. To test presentation alone, it would keep the facts, practice and feedback identical. In a bilingual system, each language should be tested with its own readers, rather than assuming that translation preserves the effect.

If the real task is making safe decisions with the tool at hand, those decisions should be measured too. Enjoyment and speed belong in the evaluation as well. They simply cannot stand in for the outcome we care about.

It is tempting to describe all this as a widening gap between machine intelligence and what a human can absorb. I find that a useful way to think, and I have said why I suspect it. But it stays a hypothesis until model capability and human understanding are measured side by side, over time.

Understanding is not a file that can be downloaded into a mind. A reader builds it from prior knowledge, attention, attempts, mistakes, correction and time. AI can support that process. It cannot prove it has succeeded just by producing more words.

The direction I want is an AI that helps a person find the central idea, see its connections, test a consequence and get back to the evidence. Sometimes that will be a few careful paragraphs. Sometimes it will be a picture like the 1,000 calls, a conversation, a simulation, or a mix of them.

These are old ambitions in teaching and design. AI gives us new ways to pursue them at scale, and new ways to mistake persuasive output for success.

The next big advance may come from treating human understanding as something to build for, rather than something we assume will follow a good answer.