用双层优化自动筛选高质量合成对话数据，提升大模型多轮对话能力。
Bilevel Optimization of Synthetic Trajectories for Multi-Turn LLM Fine-Tuning
- 内层训练模型，外层优化权重分配，动态调整合成数据重要性。
- 在多个任务上超越基线，显著提升多轮对话性能。
- 无需人工标注，适合大规模自动生成与训练数据优化场景。
尽管大语言模型在单轮生成中表现优异，但在长时序多轮交互中仍存在困难。离线强化学习提供了一种可扩展的解决方案，但其性能高度依赖多轮轨迹数据的质量与可用性。常见做法是使用大模型或模拟器生成合成轨迹来扩充训练数据，但合成数据质量参差不齐，若将所有轨迹视为等价信息源，反而会损害模型表现。本文提出 BOOST 框架，采用双层优化：内层基于加权数据训练语言模型，外层通过少量真实验证任务训练一个轻量级重加权头，为每条轨迹分配连续权重，无需外部评分器。我们从理论上推导出一个 PAC-Bayesian 边界，揭示了三重权衡：合成数据增加多样性但可能引入任务偏移，集中权重于高质量轨迹可提升实证性能，但会减少有效样本量。实验表明，该方法持续优于多个基线。分析显示，它能有效提升与真实数据分布一致且质量更高的合成轨迹权重。
原文摘要 · Abstract (English)
While LLMs excel at single-turn generation, they struggle with long-horizon, multi-turn interactions. Offline reinforcement learning (RL) offers a scalable approach, yet its performance hinges on the availability and quality of multi-turn trajectory data. A common remedy is to augment training with synthetic trajectories generated by LLMs or simulators, but synthetic data is highly heterogeneous in quality, and naively treating all trajectories as equally informative can degrade performance. We propose BOOST, a bilevel optimization framework where the inner level trains the LLM on reweighted data and the outer level trains a lightweight reweighting head on held-out real validation tasks, assigning continuous trajectory-level weights without requiring an external judge. To ground this approach, we derive a PAC-Bayesian bound revealing a three-way trade-off: synthetic data increases diversity but risks task-shift, while concentrating weight on high-quality trajectories improves empirical performance at the cost of effective sample size. Empirically, our method consistently outperforms multiple baselines. Analysis reveals it upweights synthetic trajectories that align with the real data distribution and exhibit higher qualitative merit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。