让多模态智能体删掉冗余推理,保留关键视觉信息,提升长期任务表现。
Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents

- 用摘要作为诊断上下文,通过反向KL散度判断哪些推理可删
- 在多个视觉工具任务上,删掉37.89%~64.58%的文本,准确率仍最高
- 适合长期决策类多模态智能体,尤其关注视觉证据依赖场景
多模态大模型日益作为多步智能体部署,其显式推理虽有助于任务分解与工具协调,但会积累大量自生成文本。长期运行中,这些文本占据上下文主导地位,压制视觉证据,形成文本债务。我们观察到:一旦任务相关视觉证据被定位,后续推理即成冗余;而未被证实的假设若残留,则可能误导后续推断。因此,剪枝需剔除冗余文本,同时保留视觉证据。本文提出SPARE框架,基于Kullback-Leibler散度引导,在多模态工具使用智能体中剪枝累积推理。该方法采用紧凑的任务状态摘要作为特权诊断上下文,对每个候选段落,分别在原始上下文与摘要条件下的模型进行重播。通过有策略自我蒸馏(OPSD)计算反向KL散度,检验摘要是否充分覆盖该段而不破坏未来推理。进一步通过监督微调(SFT)优化摘要器,实现更紧凑的摘要、更广覆盖和更激进的剪枝。在多个多步视觉工具使用基准测试中,SPARE在剪枝方法中取得最高平均准确率,同时移除37.89%–64.58%的推理标记。这一优异的准确性-上下文权衡表明,减少文本主导性有助于恢复对视觉证据的依赖,并缓解对自生成语言的过度依赖。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are increasingly deployed as multi-step agents, where explicit reasoning supports task decomposition and tool coordination but also accumulates self-generated text. Over long trajectories, this text can dominate the context and suppress visual evidence, creating textual debt. We observe that reasoning becomes redundant once task-relevant visual evidence is grounded, while stale hypotheses can misguide later inference when grounding remains uncertain. Pruning must therefore remove redundant text without discarding visual evidence. We propose SPARE, a Kullback-Leibler (KL)-guided framework for pruning accumulated reasoning in multimodal tool-use agents. SPARE uses a compact task-state summary as privileged diagnostic context. For each candidate segment, it replays the same model under the original and summary-conditioned contexts. Reverse-KL divergence from on-policy self-distillation (OPSD) then tests whether the summary sufficiently covers the segment without disrupting future reasoning. We further fine-tune the summarizer with supervised fine-tuning (SFT), enabling more compact summaries, broader coverage, and more aggressive pruning. Across multi-step visual tool-use benchmarks, SPARE achieves the highest average accuracy among pruning methods while removing 37.89-64.58\% of reasoning tokens. This favorable accuracy-context trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。