I spoke to seven physics experts grappling with the AI revolution

Understanding AI
Read full story· 19 min
Physicist Suvrat Raju and colleagues created a benchmark of unpublished physics problems to test AI models. GPT-6 Astra solved nearly 90% correctly, even identifying errors in human solutions, showing rapid AI progress in physics problem-solving.

What happened

Physicist Suvrat Raju and his colleagues developed a new benchmark of unpublished theoretical physics problems to evaluate the capabilities of advanced AI models. They collected challenging, never-before-published problems from top physicists and tested leading AI systems on them. The latest model, GPT-6 Astra, solved almost 90% of the problems correctly and even identified mistakes in some human-provided solutions. The benchmark, called TheoPhysEval, was effectively mastered by Astra before its public release.

Key details

  • GPT-6 Astra answered 49 out of 50 benchmark problems correctly.
  • Each AI solution cost less than $3 in compute resources.
  • The problems were unpublished and typically take expert physicists about a day to solve.
  • Astra found subtle errors in human-written solutions during evaluation.
  • The TheoPhysEval benchmark was created by Suvrat Raju and colleagues at the International Centre for Theoretical Sciences.

Why it matters

The ability of AI models like GPT-6 Astra to solve advanced, unpublished physics problems at a high level affects research physicists, academic institutions, and AI developers. Physicists now have access to tools that can rapidly perform complex calculations, potentially changing how they approach research and error-checking. This development may reduce the time and cost required for certain types of theoretical work, and raises new questions for educators and researchers about training and collaboration in the field.


More in AI Research