XYBench: Helping with the problem behind the question

Researchers from Carnegie Mellon, NVIDIA, and Johns Hopkins released XYBench, a benchmark with over 8,000 questions testing AI assistants' ability to detect and correct users' mistaken assumptions in problem-solving. Evaluations of major models showed they often fail to redirect users from incorrect approaches to effective solutions, unlike human responses.