视觉语言模型越接近人脑早期视觉处理，越不容易被言语操控。
Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation
- 通过人脑视觉皮层活动预测度衡量模型对真实视觉信息的忠实度。
- 早期视觉皮层对齐度越高，模型在76,800次操控测试中越少顺从错误指令。
- 该效应仅出现在大脑初级视觉区（V1-V3），对高阶区域无效。
视觉语言模型在高风险场景中应用日益广泛，但其对谄媚式操控的脆弱性仍不明确，尤其体现在内部视觉表征如何影响抗干扰能力。我们评估了12个开源视觉语言模型（参数量256M–10B，涵盖6种架构），从两个维度分析：大脑对齐度（基于自然场景数据集的fMRI响应预测，覆盖8名受试者和6个视觉皮层区域）与谄媚倾向（76,800条两轮操纵提示，分5类10级难度）。区域分析显示，早期视觉皮层（V1–V3）对齐度是谄媚行为的可靠负向预测因子（r = -0.441，BCa 95% CI [-0.740, -0.031]），所有12次留一法相关均为负，其中存在否认攻击效果最强（r = -0.597，p = 0.040）。高阶语义选择区域未见此关联，表明低层级视觉编码的忠实性可为模型提供对抗语言操控的稳定锚点。代码与数据已公开于GitHub与Hugging Face。
原文摘要 · Abstract (English)
Vision-language models are increasingly deployed in high-stakes settings, yet their susceptibility to sycophantic manipulation remains poorly understood, particularly in relation to how these models represent visual information internally. Whether models whose visual representations more closely mirror human neural processing are also more resistant to adversarial pressure is an open question with implications for both neuroscience and AI safety. We investigate this question by evaluating 12 open-weight vision-language models spanning 6 architecture families and a 40$\times$ parameter range (256M--10B) along two axes: brain alignment, measured by predicting fMRI responses from the Natural Scenes Dataset across 8 human subjects and 6 visual cortex regions of interest, and sycophancy, measured through 76,800 two-turn gaslighting prompts spanning 5 categories and 10 difficulty levels. Region-of-interest analysis reveals that alignment specifically in early visual cortex (V1--V3) is a reliable negative predictor of sycophancy ($r = -0.441$, BCa 95\% CI $[-0.740, -0.031]$), with all 12 leave-one-out correlations negative and the strongest effect for existence denial attacks ($r = -0.597$, $p = 0.040$). This anatomically specific relationship is absent in higher-order category-selective regions, suggesting that faithful low-level visual encoding provides a measurable anchor against adversarial linguistic override in vision-language models. We release our code on \href{https://github.com/aryashah2k/Gaslight-Gatekeep-Sycophantic-Manipulation}{GitHub} and dataset on \href{https://huggingface.co/datasets/aryashah00/Gaslight-Gatekeep-V1-V3}{Hugging Face}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。