Get the app

Perplexity and Intelligence

Open AI News
Read full story· 7 min
OpenAI emphasizes perplexity, a measure of prediction accuracy, as a key metric for tracking AI progress and fostering industry-wide safety standards. They share internal perplexity data to promote transparency and a scientific basis for future AI development pacing.

What happened

OpenAI announced that it is making its internal perplexity data public, aiming to use perplexity as a core metric for tracking progress in AI model development. Perplexity measures how accurately a model predicts the next token in a dataset, with lower values indicating better performance. The company shared historical perplexity figures for its major models, including GPT-3.5 and GPT-4, and explained how these metrics correspond to advances in model capabilities. OpenAI stated that this move is intended to promote transparency and help establish shared safety standards across the AI industry.

Key details

  • Perplexity is used as a standard measure of prediction accuracy in language models.
  • GPT-3.5 achieved a log-perplexity of approximately 0.99 on OpenAI's main evaluation dataset.
  • GPT-4 reached a log-perplexity of about 0.91.
  • Recent models have achieved log-perplexity below 0.8, leading to new capabilities in areas like cybersecurity and mathematics.
  • OpenAI published its internal perplexity measurements for the first time.

Why it matters

AI developers, researchers, and industry regulators now have access to OpenAI's internal perplexity data, which can serve as a benchmark for comparing model performance and tracking progress. This transparency allows for more informed discussions about AI safety standards and development pacing, giving other companies and stakeholders a clearer basis for evaluating advances and setting industry norms. The release of these metrics may influence how organizations assess risks and capabilities as AI models continue to evolve.

#model-evaluation-metrics#openai

More in LLM & Text Generation