通过结构感知的相对优化,提升排序模型的稳定性和效果。
Structure-aware Relative Policy Optimization for Ranking

- 用加权肯德尔距离衡量排序间的差异,更精细捕捉排列变化。
- 奖励改进按单位排序变化归一化,避免误判相似但结构不同的排序。
- 尤其在反馈少或目标复杂时表现更好,适合高阶排序任务。
排序是现代信息检索系统的核心组件。强化学习(RL)为直接优化整体排序列表上的粗粒度反馈和系统级目标提供了灵活框架。然而,现有基于RL的排序方法通常将每个采样排列视为原子输出,并主要通过标量奖励进行评估,忽略了不同排序列表之间的结构关系。因此,奖励相似但排列模式差异显著的排列可能获得相近的优化信号,导致信用分配不准确和策略更新过于激进。为解决这一问题,我们提出SRPO——一种面向列表级排序的结构感知相对策略优化框架。SRPO采用加权肯德尔距离度量采样排列间的差异,并将成对奖励差值按对应距离归一化,量化每单位排序变化带来的奖励提升,从而强调高效局部优化,特别是涉及顶部位置的调整。在两种排序场景下的实验表明,显式建模排列级差异能提升列表级排序的有效性与稳定性,尤其在有限反馈和复杂列表级优化设置下表现优异。
原文摘要 · Abstract (English)
Ranking is a fundamental component of modern information access systems. Reinforcement learning (RL) provides a flexible framework for directly optimizing coarse-grained feedback and system-level objectives defined over the complete ranking list. However, existing RL-based ranking methods typically treat each sampled permutation as an atomic output and evaluate it primarily through a scalar reward, overlooking the structural relationships among different ranking lists. Consequently, permutations with similar rewards but substantially different permutation patterns may receive comparable optimization signals, potentially leading to inaccurate credit assignment and overly aggressive policy updates. To address this limitation, we propose SRPO, a \textbf{S}tructure-aware \textbf{R}elative \textbf{P}olicy \textbf{O}ptimization framework for listwise ranking. SRPO measures the discrepancy between sampled permutations using a top-weighted Kendall-tau distance and normalizes their pairwise reward differences by the corresponding distances. It quantifies the reward improvement per unit of ranking change, thereby emphasizing efficient local refinements, particularly those involving top-ranked positions. Experimental results across two ranking scenarios demonstrate that explicitly modeling permutation-level differences improves the effectiveness and stability of listwise ranking, with particularly favorable performance in limited-feedback and complex list-level optimization settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。