AI is getting better faster than experts can reliably test whether new systems are safe, METR warns

The Decoder
Read full story· 2 min
METR reports that AI models are rapidly improving at research tasks, surpassing human professionals on many coding tasks and doubling task length capability every 3.6 months, raising concerns about assessing AI safety risks. Current public data may be insufficient to track this acceleration, potentially leaving humans unable to evaluate catastrophic risks from new AI systems.

What happened

The research group METR released findings that AI models are advancing quickly in performing research tasks, especially coding and computational work. These models now outperform experienced human professionals on many such tasks and are rapidly increasing the length of tasks they can handle. METR noted that the available public data does not adequately track how fast these improvements are happening. The group warned that this could make it difficult for people to assess and respond to serious safety risks from new AI systems.

Key details

  • AI models now surpass experienced human professionals on many coding and computational tasks.
  • The length of tasks AI can handle doubles every 3.6 months.
  • METR's data suggests a 50% success rate on 200-hour tasks could be reached by July 2027 if trends continue.
  • Leading models analyzed include Opus 5.5 and GPT-6 Astra.
  • AI systems still have difficulty with open-ended questions and tasks that lack immediate feedback.

Why it matters

AI developers, safety researchers, and regulators may not have reliable ways to evaluate the risks posed by rapidly advancing AI systems. This creates a situation where new models could introduce serious or catastrophic risks before humans are able to properly test or understand them. The lack of adequate public data and the limitations of current testing methods mean that organizations may not be able to keep up with the pace of AI progress when making safety decisions.


More in AI Research