无需源数据实现脑解码,解决个体差异与模态对齐难题
Probability Distribution Alignment and Low-Rank Weight Decomposition for Source-Free Domain Adaptive Brain Decoding
- 基于无源域适应框架,避免隐私泄露和存储负担
- 对齐边缘概率分布,提升图像与文本到fMRI的联合重构效果
- 低秩权重分解降低计算开销,兼顾性能与效率
脑解码面临个体差异、模态对齐和高维嵌入等挑战。现有方法依赖源被试数据,导致隐私泄露与存储压力;模态对齐仅关注softmax概率分布,忽略边缘概率分布对齐,造成模态错位;图像与文本分别与fMRI对齐,未考虑两者间的复杂交互,影响图像重建质量;同时,CLIP嵌入维度过高带来巨大计算成本。虽可通过忽略图像块数与文本标记数来降维,但会显著损害模型性能。为此,本文提出一种无源域适应脑解码框架,通过概率分布对齐与低秩权重分解,在不使用源数据的前提下,有效缓解上述问题。
原文摘要 · Abstract (English)
Brain decoding currently faces significant challenges in individual differences, modality alignment, and high-dimensional embeddings. To address individual differences, researchers often use source subject data, which leads to issues such as privacy leakage and heavy data storage burdens. In modality alignment, current works focus on aligning the softmax probability distribution but neglect the alignment of marginal probability distributions, resulting in modality misalignment. Additionally, images and text are aligned separately with fMRI without considering the complex interplay between images and text, leading to poor image reconstruction. Finally, the enormous dimensionality of CLIP embeddings causes significant computational costs. Although the dimensionality of CLIP embeddings can be reduced by ignoring the number of patches obtained from images and the number of tokens acquired from text, this comes at the cost of a significant drop in model performance, creating a dilemma. To overcome these limitations, we propose a source-free domain adaptation-based brain decoding framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。