用向量优化替代数据混合比例，大幅降低大模型持续预训练调参成本
OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
- 为每个数据集训练单独模型，提取参数变化向量，事后用贝叶斯优化找最优组合
- 在日文、中文、数学、代码任务上，性能优于传统混合与平均方法，搜索成本降低15-35倍
- 同一向量池可重优化生成定制模型，无需重新训练，适合快速适配新任务
持续预训练广泛用于将大语言模型适配至目标语言和领域，但训练数据混合比例是敏感超参数，需在训练前固定，选择不当可能浪费数周算力。本文提出 OptiMer，将比例选择与训练解耦：为每个数据集训练一个 CPT 模型，提取其分布向量（代表该数据集引起的参数偏移），通过贝叶斯优化事后搜索最优组合权重。在 Gemma 3 27B 上跨语言（日语、中文）和领域（数学、代码）的实验表明，OptiMer 始终优于数据混合与模型平均基线，搜索成本降低 15–35 倍。关键发现包括：1）优化后的权重可解释为数据混合比例，以之重新训练可提升数据混合 CPT 性能；2）同一向量池可针对特定目标重优化，无需任何重训练，按需生成定制模型。本工作证明，数据混合比例选择可从预训练决策重构为分布向量的事后优化，为持续预训练提供更灵活范式。
原文摘要 · Abstract (English)
Continual pre-training is widely used to adapt LLMs to target languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: they must be fixed before training begins, and a suboptimal choice can waste weeks of compute. In this work, we propose OptiMer, which decouples ratio selection from training: we train one CPT model per dataset, extract each model's distribution vector, which represents the parameter shift induced by that dataset, and search for optimal composition weights post-hoc via Bayesian optimization. Experiments on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code) show that OptiMer consistently outperforms data mixture and model averaging baselines with 15-35 times lower search cost. Key findings reveal that 1) the optimized weights can be interpreted as data mixture ratios, and retraining with these ratios improves data mixture CPT, and 2) the same vector pool can be re-optimized for a given objective without any retraining, producing target-tailored models on demand. Our work establishes that data mixture ratio selection, traditionally a pre-training decision, can be reformulated as a post-hoc optimization over distribution vectors, offering a more flexible paradigm for continual pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。