arXiv:2510.13312cs.CLcs.IR2025-10ACL被引 8

用强化学习让对话问答更灵活,能动态调整搜索与推理。

ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering

  • 通过强化学习在对话中交替进行搜索与推理,实现自适应行为。
  • 在五个数据集上超越现有模型,7B版本在多个指标上领先。
  • 适合需要理解意图变化的复杂对话系统研究者使用。

我们提出ChatR1,一种基于强化学习(RL)的对话问答(CQA)推理框架。在对话中,用户意图随回合演变,语句常不明确,需上下文理解、查询重写及检索与生成的动态协调。不同于静态的‘重写、检索、生成’流程,ChatR1在对话回合间交错执行搜索与推理,通过强化学习学习探索性与适应性行为。为应对强化学习中奖励稀疏和延迟的问题,我们设计了意图感知奖励,通过对齐检索与推理与不断演化的用户目标,提供每轮反馈。ChatR1在3B和7B模型基础上表现强劲,在五个CQA数据集上优于竞争模型,采用F1、BERTScore和大模型评分等多种评估指标。数据集涵盖主题转移、意图演化、混合主导对话与多文档依据,全面检验模型性能。消融实验验证了意图感知奖励的有效性。分析显示多样化的推理路径与高效检索工具利用。模型还展现出跨领域鲁棒性,表明基于强化学习的推理使系统比静态流程更具灵活性与上下文感知能力。

原文摘要 · Abstract (English)

We present ChatR1, a reasoning framework based on reinforcement learning (RL) for conversational question answering (CQA). Reasoning plays an important role in CQA, where user intent evolves across dialogue turns, and utterances are often underspecified, requiring contextual interpretation, query reformulation, and dynamic coordination between retrieval and generation. Unlike static `rewrite, retrieve, and generate' pipelines, ChatR1 interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through RL. To address the challenge of sparse and delayed rewards in RL, we propose an intent-aware reward that provides turn-level feedback by aligning retrieval and reasoning with evolving user goals. ChatR1 demonstrates strong performance on both 3B and 7B model backbones, outperforming competitive models on five CQA datasets, measured by different metrics (F1, BERTScore, and LLM-as-judge). We include a diverse set of CQA datasets to cover topic shifts, evolving intents, mixed-initiative dialogues, and multi-document grounding, testing ChatR1's performance from various aspects. Ablation studies confirm the effectiveness of the intent-aware reward. Our analyses further reveal diverse reasoning trajectories and effective use of the search tool. ChatR1 also generalizes robustly across domains, demonstrating that RL-based reasoning enables more flexible and context-aware behavior than static CQA pipelines.

对话问答强化学习推理机制检索增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。