通过图结构建模属性关系，提升视觉语言模型的置信度校准能力。
When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
- 构建属性符号图，用图注意力网络捕捉属性间依赖关系。
- 在9个基准上，平均ECE降低约37%（最优策略）和17%（次优策略）。
- 适合需要高可信度输出的视觉语言模型部署场景。
可靠置信度估计仍是视觉语言模型测试时自适应中的关键瓶颈，提示调优虽能提升零样本准确率，但常因熵驱动的过度自信导致校准性下降。现有方法使用大语言模型生成的类别属性并结合对比正则化缓解此问题，但将属性独立处理，忽略其相互关系。本文提出ARGTCA，将（类别，属性）对表示为符号属性图中的节点，通过对比目标训练图注意力网络（GAT），生成包含属性间依赖关系的结构化嵌入。设计两种属性选择策略：ARGTCA-DIV用于类内多样性，ARGTCA-DISC用于类间区分性。在九个基准上的实验表明，ARGTCA-DIV相比基线平均降低约37%的期望校准误差（ECE），ARGTCA-DISC始终位列第二，平均降低约17%的ECE。结果表明，建模符号属性交互为视觉语言模型的可靠测试时自适应提供了原则性路径。
原文摘要 · Abstract (English)
Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) using contrastive objectives to produce structurally informed embeddings that capture inter-attribute dependencies. We introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across nine benchmarks show that ARGTCA-DIV reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines, while ARGTCA-DISC consistently performs as the second-best variant, reducing average ECE by approximately ~17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。