通过动态调整数据分布，实现大模型剪枝后的跨领域性能均衡。
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization
- 基于分布鲁棒优化，动态调整训练数据分布以平衡多任务表现。
- 在单语和多语场景下，剪枝后困惑度与下游任务指标均优于同类模型。
- 可自动确定最优损失函数和数据比例，适合多场景部署应用。
大语言模型（LLMs）虽表现优异，但模型规模扩大带来计算成本上升。结构化剪枝虽能缩减模型并加速推理，但常导致不同领域性能不均，产生偏差。为此，本文提出DRPruning，通过在训练中动态调整数据分布，恢复异构多任务数据下的性能均衡。实验表明，在单语与多语设置下，DRPruning在剪枝与持续预训练后，于困惑度、下游任务及指令微调上均超越同等规模模型。进一步分析显示，该方法对各类领域和分布偏移具有强鲁棒性。此外，DRPruning可自动确定最优参考损失与数据比例，具备广泛应用潜力。代码与脚本见：https://github.com/hexuandeng/DRPruning。
原文摘要 · Abstract (English)
Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs. Structured pruning reduces model size and speeds up inference but often causes uneven degradation across domains, leading to biased performance. To address this, we propose DRPruning, a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. Experiments in monolingual and multilingual settings show that DRPruning surpasses similarly sized models in both pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. Further analysis demonstrates the robustness of DRPruning towards various domains and distribution shifts. Furthermore, DRPruning can determine optimal reference losses and data ratios automatically, suggesting potential for broader applications. Code and scripts are available at https://github.com/hexuandeng/DRPruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。