区分推理与模板令牌，提升大模型微调效果
Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
- 通过打乱输入输出组合，识别并分离推理与模板类令牌
- 新方法在多个任务上相比传统微调提升显著，最高增益达12.3%
- 适合需要精准推理能力的复杂任务微调场景
使用代理-任务数据集增强大语言模型代理能力时，现有方法通常对样本中所有令牌一视同仁。然而，我们指出不同角色的令牌——特别是推理令牌与模板令牌（如控制输出格式的令牌）——在重要性和学习复杂度上存在显著差异，需加以区分并分别处理。为此，我们提出一种新的洗牌感知判别器（SHAD），通过分析跨样本打乱输入输出组合后的可预测性差异来分类令牌：模板令牌因样本间重复性高而保持可预测性，推理令牌则不然。基于SHAD，我们提出推理突出微调（RFT）方法，在微调过程中自适应地强化推理令牌，相较于常见的监督微调（SFT）展现出显著性能提升。
原文摘要 · Abstract (English)
When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles - specifically, reasoning tokens versus boilerplate tokens (e.g., those governing output format) - differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。