通过双模型对比优化数据混合比例，提升大模型训练效果。
TANDEM: Bi-Level Data Mixture Optimization with Twin Networks
- 用两个互为参照的网络动态评估数据价值，自动调整不同领域数据权重。
- 在多种场景下显著提升模型性能，尤其在数据受限时优势明显。
- 理论严谨，适合需要精细化数据配比的研究与工业级模型训练。
大型语言模型的能力高度依赖于来自多个领域的训练数据。优化特定领域的数据混合比例可建模为双层优化问题，我们将其简化为单层带惩罚项的形式，并通过双网络方法求解：一个在主数据上训练的代理模型，和一个随附加数据动态更新的参考模型。所提出的TANDEM方法通过双模型差异衡量数据有效性，对更受益于额外数据的领域进行加权。相比以往方法，TANDEM具备理论保障且适用范围更广。此外，我们的双层视角揭示了新的研究场景，如数据受限情形和监督微调，其中优化后的混合比例能显著提升性能。大量实验验证了TANDEM在各类场景下的有效性。
原文摘要 · Abstract (English)
The capabilities of large language models (LLMs) significantly depend on training data drawn from various domains. Optimizing domain-specific mixture ratios can be modeled as a bi-level optimization problem, which we simplify into a single-level penalized form and solve with twin networks: a proxy model trained on primary data and a dynamically updated reference model trained with additional data. Our proposed method, Twin Networks for bi-level DatA mixturE optiMization (TANDEM), measures the data efficacy through the difference between the twin models and up-weights domains that benefit more from the additional data. TANDEM provides theoretical guarantees and wider applicability, compared to prior approaches. Furthermore, our bi-level perspective suggests new settings to study domain reweighting such as data-restricted scenarios and supervised fine-tuning, where optimized mixture ratios significantly improve the performance. Extensive experiments validate TANDEM's effectiveness in all scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。