Secret Dates in System Prompts Undermine Language Model Evaluation

A new study reveals that hidden current date metadata in system prompts of large language models (LLMs) causes significant variability in benchmark results, affecting reproducibility and leaderboard rankings. This effect surpasses other known factors like batch size or numerical precision and impacts tasks such as multiple-choice, math reasoning, code generation, and translation.