让大模型学会从复杂上下文中精准找证据,提升推理与多模态理解能力。
Context-Aware RL for Agentic and Multimodal LLMs

- 通过选择支持答案的上下文进行强化学习,引导模型关注关键细节。
- 在5个长序列任务上提升2.2%,12个视觉问答任务上提升1.8%。
- 适合需要精细上下文理解的智能体和多模态应用开发者。
大型语言模型在需要从长或复杂的上下文中识别出关键证据时表现不佳,例如工具轨迹中的一行代码或图像中的细微细节。我们提出 ContextRL,一种基于上下文感知的强化学习方法,通过间接辅助目标提升长时序推理和多模态性能。该方法给模型提供一个问题、一个答案和两个高度相似的上下文,奖励其选择支持该问题-答案对的上下文,从而促进细粒度对齐。我们在两个领域构建对比上下文数据:针对编码代理,使用轨迹作为上下文,通过条件过滤生成1000对;针对多模态推理,使用图像作为上下文,通过生成编辑和相似性搜索生成7000对。ContextRL在5个长时序基准上平均比标准GRPO提升2.2%,在12个多样化视觉问答基准上提升1.8%。与将相同对比上下文用作标准问答数据的增强基线相比,后者几乎无提升,说明性能增益来自所提出的上下文选择目标,而非对比数据本身。
原文摘要 · Abstract (English)
Large language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an \emph{indirect} auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query--answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.2% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query--context--answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。