arXiv:2604.12967cs.AI2026-04

用搜索轨迹自重构来训练搜索智能体,无需真实答案标签。

Cycle-Consistent Search: Question Reconstructability as a Proxy Reward for Search Agent Training

  • 通过轨迹反向重建问题,构建无监督奖励信号。
  • 在多个问答数据集上达到与有监督方法相当的性能。
  • 适合无真实答案标注的复杂搜索任务训练场景。

强化学习在复杂信息检索任务中优化搜索智能体方面展现出强大潜力。然而,现有方法主要依赖难以扩展的真实答案(gold supervision)。为此,我们提出无需真实标注的循环一致性搜索(CCS)框架,灵感来自无监督机器翻译和图像-图像转换中的循环一致性技术。核心假设是:高质量搜索轨迹能无损编码问题意图,因此应能准确重建原始问题,从而形成策略优化的奖励信号。为防止重建依赖表面词汇线索而非深层搜索过程,我们引入信息瓶颈机制,包括排除最终回复和对搜索查询进行命名实体识别(NER)掩码。这些约束迫使重建依赖于检索到的信息与结构框架,确保奖励信号反映信息充分性而非语言冗余。在多个问答基准测试中,CCS性能接近有监督基线,优于其他无金标方法。结果表明,CCS为缺乏金标数据的场景提供了可扩展的搜索智能体训练范式。

原文摘要 · Abstract (English)

Reinforcement Learning (RL) has shown strong potential for optimizing search agents in complex information retrieval tasks. However, existing approaches predominantly rely on gold supervision, such as ground-truth answers, which is difficult to scale. To address this limitation, we propose Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents, inspired by cycle-consistency techniques from unsupervised machine translation and image-to-image translation. Our key hypothesis is that an optimal search trajectory, unlike insufficient or irrelevant ones, serves as a lossless encoding of the question's intent. Consequently, a high-quality trajectory should preserve the information required to accurately reconstruct the original question, thereby inducing a reward signal for policy optimization. However, naive cycle-consistency objectives are vulnerable to information leakage, as reconstruction may rely on superficial lexical cues rather than the underlying search process. To reduce this effect, we apply information bottlenecks, including exclusion of the final response and named entity recognition (NER) masking of search queries. These constraints force reconstruction to rely on retrieved observations together with the structural scaffold, ensuring that the resulting reward signal reflects informational adequacy rather than linguistic redundancy. Experiments on question-answering benchmarks show that CCS achieves performance comparable to supervised baselines while outperforming prior methods that do not rely on gold supervision. These results suggest that CCS provides a scalable training paradigm for training search agents in settings where gold supervision is unavailable.

强化学习信息检索无监督训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。