IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining

Researchers propose IDEA Prune, an integrated pipeline combining enlarged model pretraining, pruning, and recovery under a unified learning rate schedule to improve token efficiency and performance in compressing large language models. Experiments show compressing 2.8B to 1.3B parameter models with up to 2T tokens benefits from this approach.