评测大模型在表格数据中定位答案来源的能力，发现其可信度很低。
ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution
- 测试多模态模型在不同表格格式下的答案溯源能力
- 表格文本输入时溯源准确率接近随机，远低于问答准确率
- 模型更擅长指出行而非列，图像格式表现优于文本格式
多模态大语言模型常用于回答结构化数据（如Markdown、JSON、图片中的表格）的问题。尽管模型能给出正确答案，用户仍需知道答案依据来自何处。本文研究结构化数据的归因/引用能力，即模型能否精准指出支持答案的具体行列。我们在多种表格格式和提示策略下评估多个mLLMs。结果表明，问答准确率虽中等，但归因准确率显著偏低，尤其是对JSON输入，所有模型的归因准确率均接近随机水平。模型在引用行方面表现优于列，且在文本格式上比图像格式更差。不同模型家族间也存在明显差异。总体而言，当前mLLMs在提供细粒度、可信赖的结构化数据归因方面不可靠，限制了其在需要透明性与可追溯性场景中的应用。
原文摘要 · Abstract (English)
Multimodal Large Language Models (mLLMs) are often used to answer questions in structured data such as tables in Markdown, JSON, and images. While these models can often give correct answers, users also need to know where those answers come from. In this work, we study structured data attribution/citation, which is the ability of the models to point to the specific rows and columns that support an answer. We evaluate several mLLMs across different table formats and prompting strategies. Our results show a clear gap between question answering and evidence attribution. Although question answering accuracy remains moderate, attribution accuracy is much lower, near random for JSON inputs, across all models. We also find that models are more reliable at citing rows than columns, and struggle more with textual formats than images. Finally, we observe notable differences across model families. Overall, our findings show that current mLLMs are unreliable at providing fine-grained, trustworthy attribution for structured data, which limits their usage in applications requiring transparency and traceability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。