让多模态模型学会判断自己信心高低,提升低资源场景下的可靠性。
Cross-Modal Bayesian Low-Rank Adaptation for Uncertainty-Aware Multimodal Learning

- 用跨模态注意力动态调整模型不确定性,实现自适应置信度估计。
- 在多个音频-文本任务上,性能优于传统方法且保持高效计算。
- 适合需要可靠预测的低资源多模态应用,如语音识别、视频理解。
大型预训练语言模型常通过参数高效微调(PEFT)适配下游任务,但现有方法多为确定性且单模态,在低资源多模态场景中难以处理预测不确定性与跨模态可靠性。本文提出CALIBER(上下文感知的低秩贝叶斯推理与嵌入正则化),一种面向音视频学习的不确定性感知多模态PEFT框架。CALIBER扩展了贝叶斯低秩适配,将适配器空间中的变分后验条件化于每层、每个词元的文本-音频交叉注意力。具体而言,文本生成的低秩特征关注帧级音频嵌入,生成局部声学上下文,进而调节低秩适配器空间内紧凑随机潜在矩阵的均值与方差。该设计使音频不仅作为额外特征源,更作为上下文可靠性信号,影响适配过程与置信度。通过将随机性限制在低维潜在组件,CALIBER在保持PEFT的计算效率与可扩展性的同时,实现异方差多模态不确定性估计。在多种文本与音频骨干网络上的实验表明,CALIBER始终匹配或超越仅文本的贝叶斯PEFT及传统多模态迁移学习基线,其中词元级交叉注意力带来最稳定的提升。结果表明,局部跨模态条件化是不确定性感知多模态适配的有效且轻量机制。
原文摘要 · Abstract (English)
Large pre-trained language models are increasingly adapted to downstream tasks using parameter-efficient fine-tuning (PEFT), but existing PEFT methods are typically deterministic and unimodal, making them poorly suited for low-resource multimodal settings where predictive uncertainty and cross-modal reliability both matter. We introduce CALIBER (Context-Aware Low-rank Inference with Bayesian Embedding Regularization), a multimodal uncertainty-aware PEFT framework for audio-text learning. CALIBER extends Bayesian low-rank adaptation by conditioning the variational posterior in the adapter space on per-layer, token-level text-audio cross-attention. Specifically, text-derived low-rank features attend to frame-level audio embeddings to produce localized acoustic context, which then modulates the mean and variance of a compact stochastic latent matrix within the rank-$r$ adapter space. This design treats audio not only as an additional feature source, but as a contextual reliability signal that shapes both adaptation and confidence. By confining stochasticity to a low-dimensional latent component, CALIBER retains the computational efficiency and scalability of PEFT while enabling heteroscedastic multimodal uncertainty estimation. Experimental results across diverse text and audio backbones show that CALIBER consistently matches or improves upon text-only Bayesian PEFT and conventional multimodal transfer-learning baselines, with token-level cross-attention yielding the most consistent gains. Our findings demonstrate that localized cross-modal conditioning is an effective and lightweight mechanism for uncertainty-aware multimodal adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。