Scale AI Launches Singapore Legal Reasoning Benchmark with IMDA
What happened
Scale AI and Singapore's Infocomm Media Development Authority (IMDA) released SG-LegalClarify on November 12, 2026. This benchmark tests 19 language models on 1,860 scenarios from 10 areas of Singapore law. The project evaluates both the legal accuracy of model answers and whether the models ask for missing facts before responding. SG-LegalClarify is the first research output from a collaboration formalized in April 2026.
Key details
- SG-LegalClarify covers 1,860 legal scenarios across 10 Singapore law domains.
- Nineteen language models were tested, including closed-book and web-enabled settings.
- Twenty-three lawyers authored and reviewed the questions, with a median experience of six years.
- Claude Fable 5 scored highest in closed-book testing at 69.8% of mandatory criteria met.
- Kimi K3 led web search-enabled testing at 77.8% of mandatory criteria met.
Why it matters
Government agencies and legal professionals in Singapore now have a benchmark tailored to their national legal system for evaluating AI legal assistants. This allows them to assess if AI models can accurately apply local laws and recognize when more information is needed. The results show that current models often fail to ask for all necessary facts, which can lead to incorrect legal conclusions, highlighting a concrete risk for anyone relying on AI for legal advice in Singapore.