用大模型推理缺失模态,让模型在不完整数据下也能准确评分。
LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations

- 用大模型提示引导补全缺失模态,不依赖完整训练数据
- 在三个动作质量评估数据集上显著超越现有方法
- 适合数据稀缺的多模态场景,如医疗或工业质检
现实世界中的多模态学习常因模态缺失而受阻。现有方法通常假设训练时模态完整,以提供重建监督或跨模态先验,这一假设不切实际。本文解决训练阶段模态不完整的更难问题,摒弃对‘上帝视角’完整数据的依赖。提出LIMSSR框架,将该挑战重新建模为条件序列推理任务。通过提示引导的上下文感知模态补全与多维表示融合,利用大语言模型从可用信息中推断隐含语义,无需直接重建。为缓解幻觉,引入掩码感知双路径聚合,动态校准推理不确定性。在三个动作质量评估数据集上的实验表明,LIMSSR在不依赖完整训练数据的情况下显著优于最先进方法,建立了一种新的数据高效多模态学习范式。代码已开源。
原文摘要 · Abstract (English)
Real-world multimodal learning is often hindered by missing modalities. While Incomplete Multimodal Learning (IML) has gained traction, existing methods typically rely on the unrealistic assumption of full-modal availability during training to provide reconstruction supervision or cross-modal priors. This paper tackles the more challenging setting of IML under training-time incomplete observations, which precludes reliance on a ``God's eye view'' of complete data. We propose LIMSSR (LLM-Driven Incomplete Multimodal Sequence-to-Score Reasoning), a framework that reformulates this challenge as a conditional sequence reasoning task. LIMSSR leverages the semantic reasoning capabilities of Large Language Models via Prompt-Guided Context-Aware Modality Imputation and Multidimensional Representation Fusion to infer latent semantics from available contexts without direct reconstruction. To mitigate hallucinations, we introduce a Mask-Aware Dual-Path Aggregation to dynamically calibrate inference uncertainty. Extensive experiments on three Action Quality Assessment datasets demonstrate that LIMSSR significantly outperforms state-of-the-art baselines without relying on complete training data, establishing a new paradigm for data-efficient multimodal learning. Code is available at https://github.com/XuHuangbiao/LIMSSR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。