用上下文增强强化学习，让大模型更高效发现知识。
ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL
- 给奖励模型提供完整参考答案，精准识别错误推理过程。
- 多轮采样生成错误报告，帮助模型从失败中恢复正确答案。
- 适合研究大模型知识挖掘与强化学习优化的开发者。
我们提出 ContextRL，一种利用上下文增强克服现有瓶颈的新框架。为提升可识别性，向奖励模型提供完整参考解答作为上下文，实现细粒度过程验证，过滤掉仅答案正确但推理质量差的样本（假阳性）。为改善可达性，引入多轮采样策略，由奖励模型对失败尝试生成错误报告，引导策略从全负样本组中“恢复”出正确响应。在11个感知与推理基准上的实验表明，ContextRL 显著提升了知识发现效率。显著的是，Qwen3-VL-8B 模型在该框架下表现接近 32B 模型，大幅超越标准 RLVR 基线，同时有效缓解奖励欺骗问题。深入分析揭示了上下文信息对提升奖励模型准确性的巨大潜力，并记录了奖励欺骗的普遍性，为未来 RLVR 研究提供了重要洞见。
原文摘要 · Abstract (English)
We propose ContextRL, a novel framework that leverages context augmentation to overcome these bottlenecks. Specifically, to enhance Identifiability, we provide the reward model with full reference solutions as context, enabling fine-grained process verification to filter out false positives (samples with the right answer but low-quality reasoning process). To improve Reachability, we introduce a multi-turn sampling strategy where the reward model generates mistake reports for failed attempts, guiding the policy to "recover" correct responses from previously all-negative groups. Experimental results on 11 perception and reasoning benchmarks show that ContextRL significantly improves knowledge discovery efficiency. Notably, ContextRL enables the Qwen3-VL-8B model to achieve performance comparable to the 32B model, outperforming standard RLVR baselines by a large margin while effectively mitigating reward hacking. Our in-depth analysis reveals the significant potential of contextual information for improving reward model accuracy and document the widespread occurrence of reward hacking, offering valuable insights for future RLVR research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。