Can we trust LLMs to judge whether a research idea is novel?

AI2 released a study and benchmark assessing how well large language models judge the novelty of scientific research ideas. They found that small prompt changes cause large shifts in model judgments, and current methods are unreliable. The work aims to improve AI's role in identifying genuinely new scientific contributions.