通过重放低熵激活,无需训练即可提升多模态大模型的推理能力。
Boosting Reasoning in Large Multimodal Models via Activation Replay
- 在测试时重放基础模型的低熵视觉激活,调节强化学习后的模型输出。
- 显著提升数学、视频推理等任务的准确率,缓解强化学习带来的推理覆盖狭窄问题。
- 方法简单有效,无需重新训练或复杂优化,适合部署在已有模型上。
近期,基于可验证奖励的强化学习(RLVR)被证明是提升大模型多模态推理能力的有效方法,但其内在机制仍不清晰。我们通过逻辑透镜分析发现,RLVR会意外地改变低熵输入激活,而高熵激活变化较小。控制实验表明这一现象与模型推理能力相关,提示调控低熵激活可能有益。为此,我们提出激活重放(Activation Replay),一种无需训练的简单方法:在测试阶段重放基础模型输入上下文中的低熵激活,以调节经RLVR微调后的模型表现。该方法在数学推理、o3类视觉代理和视频理解等多种场景中均显著提升推理性能,并改善了RLVR的通过率(Pass@K)及推理覆盖范围。对比实验显示,重放低熵激活优于高熵激活,且直接干预输入令牌优于跨模型操作,验证了方法有效性。代码已开源。
原文摘要 · Abstract (English)
Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach to incentivizing reasoning capability in Large Multimodal Models (LMMs), while the underlying mechanisms behind this post-training paradigm are poorly understood. We begin by exploring how input activations are affected by RLVR through the perspective of logit lens. Our systematic investigations across multiple post-trained LMMs suggest that RLVR shifts low-entropy activations unexpectedly, while high-entropy ones are less affected. We further demonstrate that such phenomena are associated with LMM reasoning by controlled experiments, suggesting a potentially beneficial role of modulating low-entropy activations. To this end, we propose Activation Replay, a novel simple yet effective training-free approach that boosts multimodal reasoning of post-trained LMMs without requiring expensive policy optimization. Our design involves manipulation of visual tokens at test time, replaying low-entropy activations from the input context of base LMMs to regulating the RLVR counterparts. Activation Replay triggers better reasoning across diverse scenarios, including mathematics, o3-like visual agents, and video reasoning. We further show that Activation Replay boosts Pass@K and mitigates narrower reasoning coverage of RLVR. Our design is compared against alternative choices, such as replaying high-entropy activations instead of low-entropy ones, or direct cross-model intervention instead of manipulating input tokens, demonstrating the superiority of our implementation. Code is publicly available at https://github.com/latentcraft/replay.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。