AI is getting better faster than experts can reliably test whether new systems are safe, METR warns

What happened
The research group METR released findings that AI models are advancing quickly in performing research tasks, especially coding and computational work. These models now outperform experienced human professionals on many such tasks and are rapidly increasing the length of tasks they can handle. METR noted that the available public data does not adequately track how fast these improvements are happening. The group warned that this could make it difficult for people to assess and respond to serious safety risks from new AI systems.
Key details
- AI models now surpass experienced human professionals on many coding and computational tasks.
- The length of tasks AI can handle doubles every 3.6 months.
- METR's data suggests a 50% success rate on 200-hour tasks could be reached by July 2027 if trends continue.
- Leading models analyzed include Opus 5.5 and GPT-6 Astra.
- AI systems still have difficulty with open-ended questions and tasks that lack immediate feedback.
Why it matters
AI developers, safety researchers, and regulators may not have reliable ways to evaluate the risks posed by rapidly advancing AI systems. This creates a situation where new models could introduce serious or catastrophic risks before humans are able to properly test or understand them. The lack of adequate public data and the limitations of current testing methods mean that organizations may not be able to keep up with the pace of AI progress when making safety decisions.
