通过增强相邻词元差异，减少大模型推理中的循环重复问题。
Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN
- 在前一词元信息基础上动态放大当前词元表示差异
- 在数学推理任务中准确率提升，循环推理率降低40%以上
- 适合需要长链推理的模型微调，尤其适配低秩优化
近期如OpenAI-o1和DeepSeek-R1等模型在复杂推理任务中展现出卓越性能，依赖于长链推理（Long-CoT）。尽管将此能力蒸馏至学生模型可显著提升表现，但本文发现：使用全参数微调或低秩适配（LoRA）在长链推理数据上训练时，模型常出现循环推理现象，即反复重复先前推理步骤直至达到最大长度限制。进一步分析表明，相邻词元表示差异较小与循环推理倾向正相关。为此，本文提出一种新结构——移位前馈网络（Shift-FFN），在输入前馈网络前，用前一词元信息编辑当前词元表示，从而动态放大相邻词元间的表示差异。在多个数学推理任务上的大量实验表明，结合LoRA与Shift-FFN的方法，在不同数据规模下均实现更高准确率及更低的循环推理率，优于全参数微调与标准LoRA。代码与数据已公开于https://anonymous.4open.science/r/Shift-FFN。
原文摘要 · Abstract (English)
Recently, models such as OpenAI-o1 and DeepSeek-R1 have demonstrated remarkable performance on complex reasoning tasks through Long Chain-of-Thought (Long-CoT) reasoning. Although distilling this capability into student models significantly enhances their performance, this paper finds that fine-tuning LLMs with full parameters or LoRA with a low rank on long CoT data often leads to Cyclical Reasoning, where models repeatedly reiterate previous inference steps until the maximum length limit. Further analysis reveals that smaller differences in representations between adjacent tokens correlates with a higher tendency toward Cyclical Reasoning. To mitigate this issue, this paper proposes Shift Feedforward Networks (Shift-FFN), a novel approach that edits the current token's representation with the previous one before inputting it to FFN. This architecture dynamically amplifies the representation differences between adjacent tokens. Extensive experiments on multiple mathematical reasoning tasks demonstrate that LoRA combined with Shift-FFN achieves higher accuracy and a lower rate of Cyclical Reasoning across various data sizes compared to full fine-tuning and standard LoRA. Our data and code are available at https://anonymous.4open.science/r/Shift-FFN
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。