用强化学习提升条件语义相似度判断,让大模型更懂细微语义差异。
PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similarity
- 分两阶段训练:先学基础打分,再融合多种目标优化排序能力。
- 在基准上达到48.18的斯皮尔曼相关系数,刷新交叉编码器性能纪录。
- 提出并行切片排序奖励机制,精准分配每个生成结果的奖励信号。
条件语义文本相似度(C-STS)在特定条件下衡量文本段落间的语义接近程度,克服了传统STS的模糊性。现有方法多为判别式模型,未能充分运用大语言模型(LLMs)和强化学习(RL)的最新进展。尽管RL适合此任务,因可直接优化非可微的斯皮尔曼排名指标并引导推理过程,但直接应用列表式强化学习效果不佳,因模型难以处理复杂粗粒度奖励信号。为此,我们提出PoLi-RL,一种新颖的点对列表强化学习框架。该框架采用两阶段课程:首先使用简单点式奖励训练模型建立基础评分能力;随后过渡到结合点式、成对与列表式目标的混合奖励,以精炼模型识别细微语义差异的能力。关键创新在于提出并行切片排序奖励(PSRR)机制,通过并行计算同一索引下不同样本的完成项的排名奖励,为每个生成结果提供精确区分的学习信号,实现细粒度信用分配与有效优化。在官方C-STS基准上,PoLi-RL取得48.18的斯皮尔曼相关系数,成为交叉编码器架构的新最佳表现。作为首个成功将强化学习应用于C-STS的工作,本研究为对齐大模型完成复杂、基于排名的条件判断任务提供了强大范式。
原文摘要 · Abstract (English)
Conditional Semantic Textual Similarity (C-STS) measures the semantic proximity between text segments under a specific condition, thereby overcoming the ambiguity inherent in traditional STS. However, existing methods are largely confined to discriminative models, failing to fully leverage recent breakthroughs in the NLP community involving Large Language Models (LLMs) and Reinforcement Learning (RL). RL is a particularly well-suited paradigm for this task, as it can directly optimize the non-differentiable Spearman ranking metric and guide the reasoning process required by C-STS. Nevertheless, we find that naively applying listwise RL fails to produce meaningful improvements, as the model struggles with complex, coarse-grained reward signals, leading to optimization difficulties. To address this challenge, we introduce PoLi-RL, a novel Point-to-List Reinforcement Learning framework. PoLi-RL employs a two-stage curriculum: it first trains the model with a simple pointwise reward to establish fundamental scoring capabilities, then transitions to a hybrid reward that combines pointwise, pairwise, and listwise objectives to refine the model's ability to discern subtle semantic distinctions. Crucially, we propose an innovative Parallel Slice Ranking Reward (PSRR) mechanism that computes ranking rewards in parallel slices, where each slice consists of completions with the same index from different samples. This provides a precise, differentiated learning signal for each individual completion, enabling granular credit assignment and effective optimization. On the official C-STS benchmark, PoLi-RL achieves a Spearman correlation coefficient of 48.18, establishing a new SOTA for the cross-encoder architecture. As the first work to successfully apply RL to C-STS, our study introduces a powerful paradigm for aligning LLMs for complex, ranking-based conditional judgment tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。