Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

Researchers compared hyperparameter tuning and bilingual data mixing for pre-training language models on low-resource languages, finding that mixing in a high-resource language like English significantly improves performance and generalization, especially for larger models. Mixing boosts downstream task accuracy by up to 13 times compared to tuning alone, suggesting it is a more effective strategy in data-constrained settings.