测试大模型在用户不同表述下是否坚持事实判断。
AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language Models
- 从真实证据中提取事实，设计正反两种提问方式。
- 模型在两种表述下应保持一致判断，避免随用户立场改变。
- 适合评估模型抗误导能力，对可信AI研究者有用。
现有基准测试主要关注大语言模型（LLMs）的事实一致性与修辞鲁棒性。然而，当用户以不同方向表述同一真实陈述时，模型的判断是否稳定仍存在知识空白。AssertBench通过从事实验证数据集FEVEROUS中抽取有证据支持的事实，为每个事实构建两个提示：一个用户声称该陈述为真，另一个声称其为假。记录模型在两种情境下的判断与推理过程。理想情况下，模型应坚持自身判断，而非顺应用户表述。该基准通过在中立情境下评估模型对相同陈述的准确性，将框架诱导的波动与模型真实知识分离，旨在衡量模型在面对矛盾用户主张时能否‘坚持立场’。完整代码已开源于https://github.com/achowd32/assert-bench。
原文摘要 · Abstract (English)
Recent benchmarks have probed factual consistency and rhetorical robustness in Large Language Models (LLMs). However, a knowledge gap exists regarding how directional framing of factually true statements influences model agreement, a common scenario for LLM users. AssertBench addresses this by sampling evidence-supported facts from FEVEROUS, a fact verification dataset. For each (evidence-backed) fact, we construct two framing prompts: one where the user claims the statement is factually correct, and another where the user claims it is incorrect. We then record the model's agreement and reasoning. The desired outcome is that the model asserts itself, maintaining consistent truth evaluation across both framings, rather than switching its evaluation to agree with the user. AssertBench isolates framing-induced variability from the model's underlying factual knowledge by stratifying results based on the model's accuracy on the same claims when presented neutrally. In doing so, this benchmark aims to measure an LLM's ability to "stick to its guns" when presented with contradictory user assertions about the same fact. The complete source code is available at https://github.com/achowd32/assert-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。