用物理模型调节学习步长，让大模型自我优化更稳定高效
Physics-Guided Policy Optimization with Self-Distillation
- 基于信息量动态调整更新步长，避免错误修正干扰训练
- 在Science-QA上4个领域中3个超越原方法，最高提升4.5分
- 适合需要稳定微调的大模型应用，尤其对抗训练崩溃场景
自蒸馏策略（SDPO）已成为大模型后训练的流行范式，但其对每步更新的信任度敏感：某些批次中自教师的修正极具价值，而另一些则可能误导模型，固定步长统一应用易导致训练不稳定。受粘性流体动力学启发，并在随机微分方程层面形式化类比，本文提出物理引导的策略优化（PGPO），引入由学生预测与反馈条件教师间互信息估计决定的信息调制步长乘子。该机制保持了标准SGD的一阶弱逼近保证，且每次迭代开销可忽略。在Science-QA数据集上的实验表明，PGPO在4个领域中的3个优于SDPO，性能提升最高达+4.5分，且在SDPO后期崩溃的设置下仍保持稳定。
原文摘要 · Abstract (English)
Self-distilled policy optimization (SDPO) has become a popular paradigm for LLM post-training, where a model learns from its own predictions conditioned on privileged information. SDPO, however, is sensitive to how much each update step should be trusted: corrections from a self-teacher can be highly informative on some batches and misleading on others, and applying them uniformly with a fixed step size can destabilize training. Drawing inspiration from viscous-fluid dynamics and formalizing the analogy at the SDE level, we propose Physics-Guided Policy Optimization (PGPO), which introduces an information-modulated step-size multiplier derived from a mutual-information estimate between the student's predictions and the feedback-conditioned teacher. We show that this modulation preserves the order-1 weak-approximation guarantees of vanilla SGD, and incurs negligible overhead per iteration. We evaluate PGPO on the Science-QA dataset, where it outperforms SDPO on 3 of the 4 domains with gains of up to +4.5 points, while remaining stable in a setting where SDPO collapses late in training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。