Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

NVIDIA and AI4Bharat with IIT Madras developed Nemotron-pretraining-Indic-v1, a dataset of 2.8 trillion tokens covering 22 Indian languages, plus an open GPU-accelerated pipeline for data generation and quality checks. This resource aims to boost Indic language AI research by providing extensive curated and synthetic training data, including native scripts and romanized forms, to help overcome data scarcity in Indian language technology.