解释了为何贪婪对齐方法在实践中如此高效
Demystifying the unreasonable effectiveness of online alignment methods
- 改用决策导向的性能评估标准
- 证明贪婪对齐方法累积损失恒定（O(1））
- 适合关注对齐方法理论机制的研究者
基于纯贪婪更新的迭代对齐方法在实践中表现极为出色，但现有理论中关于$O(\log T)$ KL正则化后悔的保证似乎与其实际性能存在差距。本文认为，这种不匹配源于后悔准则本身：KL正则化后悔将学习的统计代价与软化训练策略带来的探索性随机化混淆在一起。为分离这两种效应，我们研究了传统的温度为零的后悔准则，该准则仅评估推理时最优响应的表现。在此决策中心的性能定义下，我们证明了标准贪婪在线对齐方法（包括在线RLHF和在线DPO）实现常数（O(1)）累积后悔。通过将识别最佳响应的成本与正则化引起的随机性分离开来，我们的结果为贪婪对齐方法在实践中的卓越效率提供了更精确的理论解释。
原文摘要 · Abstract (English)
Iterative alignment methods based on purely greedy updates are remarkably effective in practice, yet existing theoretical guarantees of \(O(\log T)\) KL-regularized regret can seem pessimistic relative to their empirical performance. In this paper, we argue that this mismatch arises from the regret criterion itself: KL-regularized regret conflates the statistical cost of learning with the exploratory randomization induced by the softened training policy. To separate these effects, we study the traditional temperature-zero regret criterion, which evaluates only the top-ranked response at inference time. Under this decision-centric notion of performance, we prove that standard greedy online alignment methods, including online RLHF and online DPO, achieve constant \((O(1))\) cumulative regret. By isolating the cost of identifying the best response from the stochasticity induced by regularization, our results provide a sharper theoretical explanation for the practical superb efficiency of greedy alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。