小模型在简单推理任务中严重不自洽，大模型也难例外。
Existing LLMs Are Not Self-Consistent For Simple Tasks
- 用图结构和能量函数量化并修复推理不一致
- 小模型在直线/平面/家谱任务中一致性不足20%
- 适合关注AI可解释性与可靠性研究者
大型语言模型（LLMs）能力日益强大，但确保其决策透明可信需依赖自一致性——即内部推理无矛盾。本研究发现，即使在比较直线或平面上点、推理家谱等简单任务中，所有小型模型均表现出高度不一致性，即使是DeepSeek-R1和GPT-o4-mini等前沿模型也未实现完全自一致性。为量化并缓解此类问题，我们提出不一致度量指标，并设计两种自动化方法：基于图的方法与基于能量的方法。尽管这些修复措施带来部分改进，仍凸显自一致性在构建更可靠、可解释AI系统中的复杂性与关键作用。代码与数据已公开于https://github.com/scorpio-nova/llm-self-consistency。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have grown increasingly powerful, yet ensuring their decisions remain transparent and trustworthy requires self-consistency -- no contradictions in their internal reasoning. Our study reveals that even on simple tasks, such as comparing points on a line or a plane, or reasoning in a family tree, all smaller models are highly inconsistent, and even state-of-the-art models like DeepSeek-R1 and GPT-o4-mini are not fully self-consistent. To quantify and mitigate these inconsistencies, we introduce inconsistency metrics and propose two automated methods -- a graph-based and an energy-based approach. While these fixes provide partial improvements, they also highlight the complexity and importance of self-consistency in building more reliable and interpretable AI. The code and data are available at https://github.com/scorpio-nova/llm-self-consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。