不同模型架构影响幻觉产生方式，揭示了设计更可靠大模型的新方向。
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
- 对比自回归与循环架构对幻觉的影响，发现架构偏差显著改变幻觉模式。
- 特定架构更容易诱导事实性错误，且诱导难度与结构密切相关。
- 研究为构建通用抗幻觉机制提供实证基础，适合关注模型可靠性的人参考。
大型语言模型（LLMs）在日常生活中的普及得益于其生成能力，但也伴随幻觉风险与计算成本。一方面，模型常生成虚假或误导性信息，降低可靠性；另一方面，传统基于自注意力的架构存在计算瓶颈，促使循环模型等新架构涌现。然而，这两类问题往往被分开讨论。本文通过系统评估，探究架构中的归纳偏置如何影响幻觉倾向。结果显示，尽管幻觉是普遍现象，但其发生情境及诱发难易程度随架构而异。某些架构更易产生特定类型幻觉，且诱导条件差异显著。该研究强调需协同理解幻觉与架构的关系，并推动设计更普适的幻觉抑制方法。
原文摘要 · Abstract (English)
The growth in prominence of large language models (LLMs) in everyday life can be largely attributed to their generative abilities, yet some of this is also owed to the risks and costs associated with their use. On one front is their tendency to hallucinate false or misleading information, limiting their reliability. On another is the increasing focus on the computational limitations associated with traditional self-attention based LLMs, which has brought about new alternatives, in particular recurrent models, meant to overcome them. Yet it remains uncommon to consider these two concerns simultaneously. Do changes in architecture exacerbate/alleviate existing concerns about hallucinations? Do they affect how and where they occur? Through an extensive evaluation, we study how these architecture-based inductive biases affect the propensity to hallucinate. While hallucination remains a general phenomenon not limited to specific architectures, the situations in which they occur and the ease with which specific types of hallucinations can be induced can significantly differ based on the model architecture. These findings highlight the need for better understanding both these problems in conjunction with each other, as well as consider how to design more universal techniques for handling hallucinations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。