Looped Transformers虽能缩小内部表示与输出的差距，但依赖知识退化而非真正内省。
Loop as a Bridge: Can Looped Transformers Truly Link Representation Space and Natural Language Outputs?
- 通过迭代共享层增加计算深度，试图用循环实现模型内省
- 迭代次数增加虽缩小表示与输出差距，但内部知识质量下降
- 仅最终循环具备感知能力，无法实现持续内省，适合研究模型深层推理机制
大型语言模型（LLMs）常表现出内部知识与显式语言输出之间的差距。本文实证研究了循环变换器（Looped Transformers, LTs）——通过迭代共享层增加计算深度的架构——是否能利用其迭代特性作为内省机制来弥合这一差距。实验表明，尽管增加循环次数可缩小该差距，但部分源于表示中内部知识的退化。另一项实证分析进一步发现，当前LTs在循环过程中对表示的感知能力并未提升，仅在最终循环中显现。结果表明，虽然LTs为扩展计算深度提供了有前景的方向，但仍未能实现真正连接表示空间与自然语言输出所需的内省能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often exhibit a gap between their internal knowledge and their explicit linguistic outputs. In this report, we empirically investigate whether Looped Transformers (LTs)--architectures that increase computational depth by iterating shared layers--can bridge this gap by utilizing their iterative nature as a form of introspection. Our experiments reveal that while increasing loop iterations narrows the gap, it is partly driven by a degradation of their internal knowledge carried by representations. Moreover, another empirical analysis suggests that current LTs' ability to perceive representations does not improve across loops; it is only present in the final loop. These results suggest that while LTs offer a promising direction for scaling computational depth, they have yet to achieve the introspection required to truly link representation space and natural language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。