用视觉增量更新替代全图生成,提升多模态推理效率与精度
DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models

- 采用视觉增量机制,仅生成变化部分的紧凑令牌
- 视觉令牌减少55.6%,推理准确率提升3.3%
- 适合需要高效多模态推理的系统开发者
当前统一大模型通过文本推理和中间视觉状态进行多模态交错推理,但通常以完整图像形式生成每个视觉状态,造成大量视觉令牌冗余,并稀释对关键状态转移的监督。本文提出DeltaV,将全图生成替换为视觉更新:基于历史视觉状态,增量预测捕捉推理步骤间视觉变化的紧凑更新令牌,避免重复建模未变内容。为使每轮更新的令牌预算与视觉变化程度匹配,引入时间相似性(TSIM)路由器,当边际重建增益低于阈值时停止分配令牌。为进一步支持多样且可泛化的推理,构建了包含105万样本、覆盖44个任务领域的结构化思维链(StructCoT)数据集。实验表明,视觉更新范式平均减少55.6%的新生成视觉令牌,且重建保真度不受影响,多模态推理性能提升3.3%。基于StructCoT和大规模多模态数据训练的DeltaV-2B,在域内多模态推理评测中比更大规模开源模型高8.4%,在外部多模态推理与理解基准上优于同规模Qwen3-VL-2B达5.9%。代码、模型及StructCoT数据集将开源于https://github.com/Pengjie-W/DeltaV。
原文摘要 · Abstract (English)
Current Unified Large Multimodal Models (ULMMs) support interleaved multimodal reasoning through textual reasoning and intermediate visual states, but typically generate each visual state as a full image. This full-image generation paradigm introduces substantial visual-token redundancy and dilutes supervision on sparse yet reasoning-critical state transitions. We propose DeltaV, a ULMM that replaces full-image generation with visual updates. Conditioned on historical visual states, DeltaV incrementally predicts compact update tokens that capture the visual changes across reasoning steps, avoiding repeated modeling of unchanged content. To align the token budget of each update with the magnitude of visual change, DeltaV introduces a temporal similarity (TSIM) Router, which stops allocating tokens once the marginal reconstruction gain falls below a threshold. To support more diverse and generalizable reasoning, we further construct StructCoT, a large-scale interleaved multimodal reasoning dataset with 1.05M samples spanning 44 task domains. Experiments show that the visual-update paradigm reduces newly generated visual tokens by 55.6\% on average without compromising reconstruction fidelity, and improves multimodal reasoning by 3.3\% over full-image generation. Trained with StructCoT and large-scale multimodal data, DeltaV-2B further outperforms substantially larger open-source models by 8.4\% on in-domain multimodal reasoning evaluations and surpasses the comparable-scale Qwen3-VL-2B by 5.9\% on external multimodal reasoning and understanding benchmarks. Code, models, and StructCoT will be released at https://github.com/Pengjie-W/DeltaV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。