用循环对齐增强自回归思维链，让模型推理更长更准。
Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning
- 将思维链步骤与循环迭代对齐，加入每轮监督训练。
- 在未见数据上生成超长准确推理链，提升自回归模型表现。
- 适合需要长序列推理的复杂问题，如数学和逻辑题。
思维链（CoT）提示已成为提升语言模型推理能力的强大方法。然而，生成长且正确的思维链轨迹仍具挑战性。近期研究表明，循环变压器（Looped Transformers）具备出色的长度泛化能力，但其通用性和适应性有限，难以替代自回归方案。为更好利用循环变压器的优势，我们提出RELAY（通过循环对齐迭代推理）。具体而言，我们将思维链推理步骤与循环迭代对齐，并在训练循环变压器时施加中间监督。这种额外的逐轮监督不仅保留了循环变压器的长度泛化能力，还使其能够预测未见数据的思维链推理步骤。因此，我们利用该循环变压器生成复杂问题中超过训练长度的准确推理链，并用于微调自回归模型。大量实验表明，该方法显著提升了自回归模型性能。代码将在 https://github.com/qifanyu/RELAY 发布。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) prompting has emerged as a powerful technique for enhancing language model's reasoning capabilities. However, generating long and correct CoT trajectories is challenging. Recent studies have demonstrated that Looped Transformers possess remarkable length generalization capabilities, but their limited generality and adaptability prevent them from serving as an alternative to auto-regressive solutions. To better leverage the strengths of Looped Transformers, we propose RELAY (REasoning through Loop Alignment iterativelY). Specifically, we align the steps of Chain-of-Thought (CoT) reasoning with loop iterations and apply intermediate supervision during the training of Looped Transformers. This additional iteration-wise supervision not only preserves the Looped Transformer's ability for length generalization but also enables it to predict CoT reasoning steps for unseen data. Therefore, we leverage this Looped Transformer to generate accurate reasoning chains for complex problems that exceed the training length, which will then be used to fine-tune an auto-regressive model. We conduct extensive experiments, and the results demonstrate the effectiveness of our approach, with significant improvements in the performance of the auto-regressive model. Code will be released at https://github.com/qifanyu/RELAY.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。