针对视频模型缺模态时情绪识别失效问题,提出自适应修复框架。
TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs

- 测试时自蒸馏+费雪锚定修复,仅更新少量参数。
- 在0%-50%模态缺失下,长期适应仍保持性能稳定。
- 适合真实场景中多模态信号不全的模型部署。
大型视频-语言模型(LVLMs)在野外多模态情绪识别(ER)任务中表现优异,但实际部署常面临模态缺失或噪声问题。现有基于熵最小化或困惑度降低的SOTA测试时自适应(TTA)方法不适用于自回归式LVLMs,而检索增强生成(RAG)在弱观察模态下性能下降。由于缺乏真值监督,单一适应过程易积累偏差。本文提出测试时自蒸馏(TTSD)框架,利用冻结教师模型指导低秩学生模型进行参数高效更新;通过费雪锚定修复(FAR)监控费雪信息稳定性,检测分布偏移并恢复学生模型至教师锚点。在MELD、DFEW和BAH数据集上,0%-50%模态缺失条件下,该框架持续优于熵基、RAG及困惑度生成方法,在长周期适应中无性能衰减,而基线模型显著退化。
原文摘要 · Abstract (English)
Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild. ER is inherently multimodal, requiring a joint understanding of facial expressions, vocalizations, language, biosignals, and gestures. However, real-world deployment remains challenging: modalities may be missing or noisy at test time. Partial observations can be viewed as a distribution shift relative to the complete-modality distribution. SOTA TTA methods based on entropy minimization or perplexity reduction do not transfer to autoregressive LVLMs, while retrieval augmented generation (RAG) degrades when the observed modality is weak. Because no ground-truth supervision exists to verify individual updates, adaptation across this stream risks accumulating drift and degrading once the model departs from a reliable solution. An effective solution must therefore adapt to arbitrary missing-modality patterns and remain effective during continual adaptation. We address both jointly with Test-Time Self-Distillation (TTSD), a parameter-efficient framework in which a frozen teacher, trained on complete modalities, guides an adaptive low-rank student via self-distillation, updating only a negligible number of parameters. Stability is built into this same loop through Fisher-Anchored Restoration (FAR), which monitors Fisher information stability to detect convergence versus drift and restores the student toward the teacher's anchor when distributional shifts are identified. Our experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation over long adaptation horizons, where baselines without restoration progressively degrade while TTSD-FAR remains consistent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。