Combatting adversarial distillation

Open AI News
Read full story· 11 min
OpenAI has enhanced its defenses against adversarial distillation, a technique where attackers train new models on outputs from frontier AI models to bypass safeguards. Updated protections in GPT-6.2 Astra and Sol drastically reduce the risk of extracting sensitive internal reasoning, improving AI safety.

What happened

OpenAI has introduced stronger protections in its GPT-6.2 Astra and Sol models to counter adversarial distillation, a method where attackers train new models using outputs from advanced AI systems in order to bypass built-in safeguards. The company updated its defenses to specifically prevent extraction of internal reasoning steps, which are valuable for training unauthorized models. Internal evaluations showed that these new safeguards sharply reduced the success rate of such attacks. The changes were implemented as part of OpenAI’s ongoing efforts to improve the safety of its AI models.

Key details

  • Adversarial distillation attack success rate dropped from 54.0% to 0.2% after safeguards were updated in GPT-6.2 Astra.
  • The protections focus on preventing extraction of 'chain-of-thought' reasoning traces.
  • OpenAI uses adversarial testing, model training, classifiers, and API-level protections to defend against these attacks.
  • The updated safeguards align with those used for sensitive areas like biology and cybersecurity.
  • Internal studies showed that models trained on exact reasoning traces gained more advanced capabilities than those trained on summaries.

Why it matters

Organizations and developers using OpenAI’s models now face a lower risk that attackers will extract sensitive reasoning steps and use them to build unsafeguarded AI systems. This reduces the chance that advanced AI capabilities, especially in areas like cybersecurity, could be reproduced and misused by unauthorized parties. The new safeguards mean customers and partners can rely on improved protection of proprietary model behaviors and internal logic.

More on this story

  1. Why OpenAI’s latest model is obsessed with bells
    · IBM Blog
  2. OpenAI’s GPT-6.2 Astra and Sol arrive in GitHub Copilot with smarter coding and better code reviews
    · 15 sources
  3. OpenAI reassigns robotics researchers to other teams as general-purpose models might make specialized robotics AI unnecessary
    · The Decoder
  4. This AI ‘mechanic’ can peer inside the black box of other neural networks
    · Nature
  5. Astra covertly underperforming on safety research is concerning, even though it’s unlikely to be motivated by ambitious goals
    · LessWrong
  6. GPT-6 Astra Covertly Underperforms on Safety Research Tasks
    · 2 sources

More in Machine Learning