arXiv:2607.00302cs.CVcs.MM2026-07

让多模态大模型学会触觉,不丢原有视觉能力

Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

论文配图:Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs
图 1 · 摘自论文原文
  • 用掩码隔离参数空间,只更新无用部分
  • 触觉推理性能达顶尖水平,零额外计算开销
  • 适合需要新增感官又不想重训的模型应用

触觉能提供感知材料固有属性(如摩擦力、柔韧性)的物理基础,这是视觉单独难以实现的。当前将触觉引入多模态大模型的努力面临两难:紧凑模型参数有限,只能在新增触觉能力与保留原有视觉语言推理间二选一。本文提出Splash框架,通过量化预训练参数重要性,将参数空间划分为休眠和关键子空间。冻结的关键子空间作为稳定锚点,保护通用视觉知识;仅更新隔离的休眠子空间以对齐触觉信息至大语言模型。这种选择性、非破坏性扩展有效防止灾难性遗忘,确保模态扩展不损原有能力。大量实验表明,Splash在无需增加大模型推理开销的前提下,实现了出色的触觉推理性能,在SSVTP、TVL和TacQuad等基准上达到最先进水平,同时保持原有通用能力。

原文摘要 · Abstract (English)

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.

多模态触觉理解参数高效大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。