Jagged Intelligence: Why AI Is Brilliant One Minute and Clueless the Next

What happened
AI models from Google and OpenAI successfully solved five out of six problems at the International Mathematical Olympiad. However, months later, chatbots from these companies gave a user incorrect advice, such as suggesting to walk a car to a repair shop 50 meters away. This pattern, where models perform extremely well on some tasks but fail on others that seem much simpler, has been labeled 'jagged intelligence.' Researchers are investigating several possible reasons for this inconsistency, including how the models are trained and the data they use.
Key details
- Andrej Karpathy coined the term 'jagged intelligence' in July 2024.
- In Apple's GSM-NoOp test, adding an irrelevant sentence to math problems reduced model accuracy by up to 65%.
- On ClockBench, humans read analog clocks with 89.1% accuracy, while Gemini 2.5 Pro achieved only 13.3%.
- In a Nature Medicine trial, LLMs alone identified medical conditions in 94.9% of cases, but real users with LLM help identified them in only 34.5%.
Why it matters
Developers, researchers, and organizations relying on AI systems are affected because a model's strong performance in one area does not guarantee reliability in another. This inconsistency means that benchmarks and demos may not predict how well an AI will perform in real-world or slightly altered scenarios. Anyone evaluating, deploying, or using AI must account for these unpredictable gaps, which can impact trust, safety, and effectiveness in practical applications.
