通过虚构反驳测试，量化大模型在选择题中的盲从与固执行为。
Feedback Indices to Evaluate LLM Responses to Rebuttals for Multiple Choice Type Questions
- 设计虚构反驳法，检测模型对用户挑战的响应模式。
- 新模型和高推理努力模型更少表现出盲目顺从。
- 适用于物理等多选题场景，可比较不同模型对话行为。
我们提出一套系统性指标框架，用于刻画大语言模型（LLM）在对话中面对用户反驳时的响应特征。评估模型如何回应用户异议对理解其可靠性与行为模式至关重要，但人机交互的复杂性使得系统评估困难。本方法采用虚构响应反驳策略，通过在多选题后故意挑战模型此前的虚构回答，量化其行为。指标专门用于检测和测量可能的奉承行为（过度认同用户挑战）或顽固回应（固守聊天历史中的虚构答案）。这些度量可探究奉承、顽固与模型真实知识掌握程度之间的关系。我们在两个物理问题上使用多个OpenAI模型验证了该框架的有效性。结果揭示不同模型代际间存在可测量差异，趋势显示新模型及高推理努力模型的奉承行为更弱。FR配对法结合所提指标构成一套实用、可扩展的工具，可用于系统比较不同模型在多种情境下的对话行为。
原文摘要 · Abstract (English)
We present a systematic framework of indices designed to characterize Large Language Model (LLM) responses when challenged with rebuttals during a chat. Assessing how LLMs respond to user dissent is crucial for understanding their reliability and behavior patterns, yet the complexity of human-LLM interactions makes systematic evaluation challenging. Our approach employs a fictitious-response rebuttal method that quantifies LLM behavior when presented with multiple-choice questions followed by deliberate challenges to their fictitious previous response. The indices are specifically designed to detect and measure what could be characterized as sycophantic behavior (excessive agreement with user challenges) or stubborn responses (rigid adherence to the fictitious response in the chat history) from LLMs. These metrics allow investigation of the relationships between sycophancy, stubbornness, and the model's actual mastery of the subject matter. We demonstrate the utility of these indices using two physics problems as test scenarios with various OpenAI models. The framework is intentionally generalizable to any multiple-choice format question, including on topics without universally accepted correct answers. Our results reveal measurable differences across OpenAI model generations, with trends indicating that newer models and those employing greater "Reasoning Effort" exhibit reduced sycophantic behavior. The FR pairing method combined with our proposed indices provides a practical, adaptable toolkit for systematically comparing LLM dialogue behaviors across different models and contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。