模型的最终输出可能泄露图像中隐藏的敏感信息。
What do your logits know? (The answer may surprise you!)
- 通过分析视觉语言模型不同层级的表示，发现顶层logits也能暴露信息
- 即使只看最高概率的logit值，也能泄露与任务无关的图像细节
- 对模型安全有警示意义，适合关注模型隐私的研究者
近期研究表明，探测模型内部可揭示生成结果之外的丰富信息。这带来了无意或恶意的信息泄露风险，即用户可能获取模型所有者认为不可访问的内容。本文以视觉语言模型为实验平台，首次系统性比较了信息在不同“表征层级”上的保留情况，这些信息从残差流中丰富的编码内容，经由两个自然瓶颈压缩：使用调优透镜得到的残差流低维投影，以及影响模型答案的最终top-k logits。结果显示，即使仅通过模型最易获取的top logit值，也可能泄露基于图像查询中的任务无关信息，某些情况下其泄露程度甚至接近直接投影完整残差流。
原文摘要 · Abstract (English)
Recent work has shown that probing model internals can reveal a wealth of information not apparent from the model generations. This poses the risk of unintentional or malicious information leakage, where model users are able to learn information that the model owner assumed was inaccessible. Using vision-language models as a testbed, we present the first systematic comparison of information retained at different "representational levels'' as it is compressed from the rich information encoded in the residual stream through two natural bottlenecks: low-dimensional projections of the residual stream obtained using tuned lens, and the final top-k logits most likely to impact model's answer. We show that even easily accessible bottlenecks defined by the model's top logit values can leak task-irrelevant information present in an image-based query, in some cases revealing as much information as direct projections of the full residual stream.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。