Perplexity and Intelligence

What happened
OpenAI announced that it is making its internal perplexity data public, aiming to use perplexity as a core metric for tracking progress in AI model development. Perplexity measures how accurately a model predicts the next token in a dataset, with lower values indicating better performance. The company shared historical perplexity figures for its major models, including GPT-3.5 and GPT-4, and explained how these metrics correspond to advances in model capabilities. OpenAI stated that this move is intended to promote transparency and help establish shared safety standards across the AI industry.
Key details
- Perplexity is used as a standard measure of prediction accuracy in language models.
- GPT-3.5 achieved a log-perplexity of approximately 0.99 on OpenAI's main evaluation dataset.
- GPT-4 reached a log-perplexity of about 0.91.
- Recent models have achieved log-perplexity below 0.8, leading to new capabilities in areas like cybersecurity and mathematics.
- OpenAI published its internal perplexity measurements for the first time.
Why it matters
AI developers, researchers, and industry regulators now have access to OpenAI's internal perplexity data, which can serve as a benchmark for comparing model performance and tracking progress. This transparency allows for more informed discussions about AI safety standards and development pacing, giving other companies and stakeholders a clearer basis for evaluating advances and setting industry norms. The release of these metrics may influence how organizations assess risks and capabilities as AI models continue to evolve.
