首次系统评估大音频语言模型的成员推理攻击，发现声音与文本绑定才会导致记忆泄露。
Membership Inference Attacks against Large Audio Language Models
- 用文本、频谱和语调特征做无模型基线，发现数据集本身就有明显可分性
- 在匹配分布的数据上测试，攻击成功率接近100%，证明记忆来自跨模态绑定
- 适用于关注隐私安全的研究者，尤其关心语音模型是否泄露训练数据
我们首次系统评估了大音频语言模型（LALMs）的成员推理攻击（MIA）。基于文本、频谱和语调特征的多模态无模型基线表明，常见音频数据集即使不依赖模型推理也呈现近乎完美的训练/测试可分性（AUC ~ 1.0），说明MIA可能主要检测的是分布偏移。因此我们提出一种无模型基线协议以控制该混淆因素。在此协议下，我们发现分布匹配的数据集能实现可靠的MIA评估，且无分布偏移伪影。我们在这些数据集上基准测试多种MIA方法，并进行模态解耦实验。结果表明，LALM的记忆是跨模态的，仅当说话人声纹与其文本绑定时才出现。这些发现为超越虚假相关性的LALMs审计建立了原则性标准。代码库已公开于 https://github.com/snooow1029/ALM_MIA。
原文摘要 · Abstract (English)
We present the first systematic Membership Inference Attack (MIA) evaluation of LALMs. Using Multi-modal Blind Baselines based on textual, spectral and prosodic features, we demonstrate that common audio datasets exhibit near-perfect train/test separability (AUC ~ 1.0) even without model inference, thus MIA may primarily detect distribution shift. We therefore introduce a blind-baseline protocol to control for this confound. Under this protocol, we identify that the distribution-matched datasets enable reliable MIA evaluation without distribution-shift artifacts. We benchmark multiple MIA methods and conduct modality disentanglement experiments on these datasets. The results reveal that LALM memorization is cross-modal, arising only from binding a speaker's vocal identity with its text. These findings establish a principled standard for auditing LALMs beyond spurious correlations. Our codebase is available at https://github.com/snooow1029/ALM_MIA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。