arXiv:2511.19314cs.AIcs.CL2025-11ACL被引 3

PRInTS让AI更会查信息,长流程任务也能精准判断每一步。

PRInTS: Reward Modeling for Long-Horizon Information Seeking

  • 用生成式模型同时打分和压缩上下文,多维度评估每一步质量。
  • 在多个数据集上,用小模型+PRInTS就超过大模型表现。
  • 适合需要长期推理与工具调用的AI助手开发与优化。

信息搜索是智能体的核心能力,需在长轨迹中跨步骤获取并推理工具生成的信息。然而,基于语言模型的智能体在处理此类多步任务时仍面临挑战。现有过程奖励模型(PRMs)针对短程推理设计,仅支持二元判断,难以捕捉工具交互、输出推理等复杂维度,也无法应对长周期任务中迅速增长的上下文。为此,我们提出PRInTS,一种具备双重能力的生成式PRM:(1) 基于多维度推理进行密集评分(如工具输出理解、工具调用信息量);(2) 轨迹摘要功能,在压缩上下文的同时保留关键评估信息。在FRAMES、GAIA(1-3级)及WebWalkerQA(易-难)等多个基准上的实验证明,使用PRInTS的best-of-n采样显著提升开源模型与专用智能体的信息搜索能力,其表现匹配甚至超越前沿模型,且仅需更小的主干模型,并优于其他强基线奖励模型。

原文摘要 · Abstract (English)

Information-seeking is a core capability for AI agents, requiring them to gather and reason over tool-generated information across long trajectories. However, such multi-step information-seeking tasks remain challenging for agents backed by language models. While process reward models (PRMs) can guide agents by ranking candidate steps at test-time, existing PRMs - designed for short reasoning with binary judgment - cannot capture richer dimensions of information-seeking steps, such as tool interactions and reasoning over tool outputs, nor handle the rapidly growing context in long-horizon tasks. To address these limitations, we introduce PRInTS, a generative PRM trained with dual capabilities: (1) dense scoring based on the PRM's reasoning across multiple dimensions of step quality (e.g., interpretation of tool outputs, tool call informativeness) and (2) trajectory summarization that compresses the growing context while preserving essential information for step evaluation. Extensive evaluations across FRAMES, GAIA (levels 1-3), and WebWalkerQA (easy-hard) benchmarks on multiple models reveal that best-of-n sampling with PRInTS enhances information-seeking in open-source models as well as specialized agents, matching or surpassing frontier models with a much smaller backbone agent and outperforming other strong reward modeling baselines.

信息搜索奖励建模长程推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。