AIs like ChatGPT fall apart in classic 'Stroop' psychological test — and that could stand in the way of achieving artificial general intelligence

1. AI Platforms & Assistants
2. OpenAI
3. ChatGPT

AIs like ChatGPT fall apart in classic 'Stroop' psychological test — and that could stand in the way of achieving artificial general intelligence News By Darren Allan published 4 June 2026

Executive control mechanisms are key

When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.

(Image credit: Getty Images)

• Copy link
• Facebook
• X
• Whatsapp
• Reddit
• Pinterest
• Flipboard
• Threads
• Email

Share this article 0 Join the conversation Follow us Add us as a preferred source on Google Newsletter Subscribe to our newsletter

• A new study tasked AIs with tackling the 'Stroop' test
• GPT and Claude performed very poorly compared to humans
• There are nuances here, but broadly, the researchers argue that improving this side of AIs is crucial for achieving artificial general intelligence

A freshly published study has pointed out a limitation of big-name AI models such as ChatGPT, albeit causing some controversy as the primary piece of research uses now outdated versions of those models – but there are nuances therein, and this doesn't make the findings irrelevant.

I'll go into that more shortly, but first, let's look at the study itself, which was highlighted on Reddit ('New study reveals top AI models completely fail the classic 'Stroop' psychological attention test') and published via the Oxford University Press in the journal PNAS Nexus.

The research consists of testing the so-called 'Stroop effect' with GPT-4o and Claude 3.5 Sonnet. As noted, these aren't the cutting-edge versions of those AIs (Large Language Models, or LLMs) – but they were at the time the initial study was carried out.

Latest Videos FromWatch full video here:

The Stroop effect refers to the phenomenon whereby the human brain gets confused when asked to name the color of the ink used to write a word, when that word can be the written version of another (incongruent) color in some cases. So, if the word 'red' is written in blue ink, that'll cause a slower response – or possibly a wrong response, where the viewer will accidentally say "red" rather than the actual color of the ink, which is blue.

This is because the brain is trying to juggle two different tasks – reading comprehension and color recognition – and so cognitive interference arises. Overriding the compulsion to read the word and say the color instead requires "executive control of attention," and this is what the authors were testing in the AI models. Both color-naming and word-reading were tested in shorter and longer lists of words (5, 10, 20, and 40 words).

You may like

• ChatGPT just announced it can pass the ‘how many “r”s in strawberry’ test, but users found otherwise
• ChatGPT can threaten to ‘key your car’ and get abusive with certain prompts
• Microsoft scientists find most AI models struggle with long-running tasks

The study observes: "Like humans, both LLMs [GPT and Claude] showed relatively high accuracy on the word-reading task and performed worse in the incongruent condition [where the word doesn't match the color] than in the congruent and neutral conditions for the color-naming task."

For color naming, humans maintain around 95% accuracy even in very long tests (up to an hour), but LLMs' accuracy declined very swiftly with longer word lists under the incongruent condition (mismatched color and word name). GPT-4o was 91% accurate in a five-word test, but dropped off to 57% with 10 words, and fell away completely to 22% with 20 words (and was only 15% accurate at 40 words).

Get daily insight, inspiration and deals in your inbox

Sign up for breaking news, reviews, opinion, top tech deals, and more.

Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsorsBy submitting your information you agree to the Terms & Conditions and Privacy Policy and are aged 16 or over.

Claude 3.5 Sonnet did better, staying 76% accurate at 20 words, but again fell hopelessly to 24% in the longest test of 40 words.

The authors conclude: "The significant degradation pattern of the two LLMs suggests fundamental limitations compared with human attention."

Analysis: another necessary step on the path to AGI?

(Image credit: Shutterstock / Ryzhi)

If you've scanned through the Reddit thread, you doubtless noticed that, as mentioned at the outset, there's a lot of flak fired at this study by commenters due to the usage of outdated models of GPT and Claude.

What to read next

• Everyone’s switching to Claude but the smartest AI might surprise you
• Studies show top AI models go to 'extraordinary lengths' to stay active
• Could ChatGPT suffer Firefox’s fate? — 'The risk of falling behind is growing exponentially' as rival AI tools Gemini and Claude surge while Copilot stalls

Indeed, these older LLMs are called "state of the art" at one point by the authors – and of course, as already noted, they were cutting-edge when the main study was conducted. Still, this is unfortunate phrasing that should've been updated and tweaked now that the paper has just been published (after peer review and so forth).

However, the researchers did conduct tests on GPT-5, Claude Opus 4.1, and Gemini 2.5 Pro in September 2025, although this is somewhat buried in the paper. That more recent testing found that these models offered only "slight" improvements on their predecessors, and that they still exhibited "ongoing executive attention deficiencies, consistent with our comprehensive analysis of earlier transformer models" (as did Gemini 2.5 Pro, which was a new introduction here).

Granted, a smaller sample size was used, but the researchers still argue that overall, their study reflects a fundamental limitation which is "inherent to the architectural constraints of transformer-based LLMs".

The authors note that a caveat is that GPT-5 in 'Thinking' mode can write and then run code to ensure it performs the Stroop test flawlessly – and similar functionality can be utilized by other LLMs – but this is essentially the AI (cleverly) fudging around its inadequacies. It isn't changing the way it works or reasons more broadly, of course.

The researchers note that transformer architecture innovations for LLMs are focused on enhancing memory capabilities, which fail to address the "core limitations of attention mechanisms, specifically the need for sophisticated alerting, orienting, and executive control networks to enable cognitive flexibility."

The ultimate aim is effective goal-directed behavior, and the study observes: "Future [LLM] development might benefit from implementing more sophisticated executive control systems that can handle decision conflicts through structured, goal-directed processing rather than relying solely on enhanced memory capabilities."

The authors argue that "incorporating executive control mechanisms akin to those in biological attention is crucial for achieving artificial general intelligence [AGI]."

Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.

The best laptops for all budgetsOur top picks, based on real-world testing and comparisons

➡️ Read our full guide to the best laptops1. Best overall: Apple MacBook Air 13-inch M52. Best budget: Apple MacBook Neo3. Best Windows 11 laptopMicrosoft Surface Laptop 13-inch4. Best thin and light:Lenovo Yoga Slim 9i5. Best UltrabookAsus Zenbook S 16

TOPICS AI CATEGORIES ChatGPT Claude Gemini OpenAI Darren Allan

Darren is a freelancer writing news and features for TechRadar (and occasionally T3) across a broad range of computing topics including CPUs, GPUs, various other hardware, VPNs, antivirus and more. He has written about tech for the best part of three decades, and writes books in his spare time (his debut novel - 'I Know What You Did Last Supper' - was published by Hachette UK in 2013).

View More

You must confirm your public display name before commenting

Please logout and then login again, you will then be prompted to enter your display name.

Logout Read more AI Platforms & Assistants ChatGPT just announced it can pass the ‘how many “r”s in strawberry’ test, but users found otherwise    AI Platforms & Assistants ChatGPT can threaten to ‘key your car’ and get abusive with certain prompts    Pro Microsoft scientists find most AI models struggle with long-running tasks    AI Platforms & Assistants Everyone’s switching to Claude but the smartest AI might surprise you    AI Platforms & Assistants Studies show top AI models go to 'extraordinary lengths' to stay active    Pro Could ChatGPT suffer Firefox’s fate? — 'The risk of falling behind is growing exponentially' as rival AI tools Gemini and Claude surge while Copilot stalls    Latest in AI Platforms & Assistants AI Platforms & Assistants Meta AI's recent hack is a wake-up call for anyone who puts their trust in AI systems    Fitness Trackers Samsung Galaxy Watch users are getting a completely redesigned, AI-first app for a 'personalized experience'    Tech What to expect from Tim Cook's final WWDC — assuming he doesn't pull a Wolf of Wall Street    Computing A Californian city just became the first to permanently ban data centers    ChatGPT I used ChatGPT to enhance my resume and job hunt    Pro Microsoft reveals Scout, its first "Autopilot"    Latest in News Computing Nvidia’s new RTX Spark chip won’t come to a PC handheld soon says Huang    Gaming Accessories Belkin's new Nintendo Switch 2 Grip could solve my biggest problem with handheld mode    Speakers Ruark's new R710 Music Console supports CD, vinyl, and hi-res streaming    Gaming 'At least they're honest about it?' —Tomb Raider: Legacy of Atlantis is the latest game to come with an AI-generated content disclosure    How to Watch Football How to watch Spain vs Iraq: Free Streams & TV Channels for World Cup 2026 warm-up match    Entertainment Tip Toe full episode release date on Channel 4    LATEST ARTICLES

1. 1Many US tech firms are turning to China's DeepSeek as the bill for homegrown AI bites – American AI companies could learn a thing or two
2. 2OpenAI’s Codex helps discover HTTP/2 Bomb DoS attack that can nuke over 30GB of RAM within seconds, knocking web servers offline before they can react
3. 3With GTA 6 Online, Rockstar has a chance to free players from its greatest vehicular mistake
4. 4I'm done with multitasking on a single PC, and I'm tired of waiting for Valve — so I built a custom Steam Machine, and here's what it has changed for me
5. 55 things to expect at WWDC 2026 — from Siri 2.0 to Tim Cook's Apple farewell