构建去歧义基准,评估多模态模型的联想能力
AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity
- 拆解关联任务中的内生与外在歧义,设计混合计算评估方法
- 发现认知能力与联想能力呈强正相关,歧义导致模型行为更随机
- 为多模态大模型创造力评估提供更可靠标准,适合模型评测研究者
多模态大语言模型(MLLMs)的进展为实现通用人工智能(AGI)带来了希望。其中,创造力是AGI所需的关键能力,而联想是其基础。现有评估框架常忽略关联任务固有的歧义性,该歧义源于关联本身的多样性,影响评估可靠性。为此,我们区分出内生歧义与外在歧义,提出AssoCiAm基准,通过混合计算方法规避歧义。在MLLM上进行广泛实验,发现认知能力与联想能力存在强正相关;同时,评估中若含歧义,模型行为趋于随机。验证表明,该方法能提升评估的准确性与可靠性。
原文摘要 · Abstract (English)
Recent advancements in multimodal large language models (MLLMs) have garnered significant attention, offering a promising pathway toward artificial general intelligence (AGI). Among the essential capabilities required for AGI, creativity has emerged as a critical trait for MLLMs, with association serving as its foundation. Association reflects a model' s ability to think creatively, making it vital to evaluate and understand. While several frameworks have been proposed to assess associative ability, they often overlook the inherent ambiguity in association tasks, which arises from the divergent nature of associations and undermines the reliability of evaluations. To address this issue, we decompose ambiguity into two types-internal ambiguity and external ambiguity-and introduce AssoCiAm, a benchmark designed to evaluate associative ability while circumventing the ambiguity through a hybrid computational method. We then conduct extensive experiments on MLLMs, revealing a strong positive correlation between cognition and association. Additionally, we observe that the presence of ambiguity in the evaluation process causes MLLMs' behavior to become more random-like. Finally, we validate the effectiveness of our method in ensuring more accurate and reliable evaluations. See Project Page for the data and codes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。