The Importance Of AI Onboarding: AI Doesn’t Know Your Churn Definition

What happened
Anthropic tested its Claude AI model on internal analytics tasks and found that, without specific business context, the model's accuracy was only 21%. When the company provided curated business context, the model's accuracy increased to over 95%. This experiment was conducted using Anthropic's own data warehouse and analytics processes. The results were shared as part of Anthropic's efforts to improve self-service analytics with AI tools.
Key details
- Claude AI's accuracy on analytics tasks was 21% without added context.
- Accuracy rose above 95% after providing curated business context.
- The test involved Anthropic's internal data warehouse and analytics evaluations.
- AI models can generate incorrect or generic answers without clear metric definitions.
- Anthropic recommends documenting key performance indicators and metric definitions for AI onboarding.
Why it matters
Companies using AI for analytics need to provide detailed business context to achieve reliable results. Data teams will need to invest time in documenting metrics, definitions, and data relationships to ensure AI models deliver accurate answers. Employees and executives must adjust their processes to include onboarding steps for AI tools, or risk receiving misleading or incorrect analytics.
