给大模型生成的每个词打分，提升训练效果并防止遗忘。
AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training
- 分离适应性与稳定性，按生成路径评估每个词的价值。
- 在不依赖保留数据的情况下，用代理方法近似保持预训练能力。
- 适用于大模型微调和偏好优化，能减少无效训练开销。
令牌选择对大模型后训练至关重要。现有方法多依赖局部启发式规则，很少将令牌选择形式化为对响应令牌的合理估值。我们提出 AlphaToken，一个将估值解耦为适应性（促进目标任务学习）与稳定性（保留预训练能力）的框架，并通过结合局部梯度的直接路径信号与自回归生成中的下游因果路径信号，实现路径感知。由于保留数据通常不可用，AlphaToken 在预训练参考模型基础上，使用 Fisher-drift 代理近似稳定性。为高效计算，我们扩展 Ghost Dot-Product 至令牌级估值。在微调和偏好优化中，AlphaToken 掩码低价值令牌，使训练信号聚焦于更有价值的位置。实验表明，该方法提升了后训练性能并缓解了灾难性遗忘。
原文摘要 · Abstract (English)
Token selection is pivotal for effective LLM post-training. However, existing methods mostly rely on local heuristics and rarely formulate token selection as a principled valuation of individual response tokens. We introduce $\textbf{AlphaToken}$, a response token valuation framework that decouples valuation into $\textbf{adaptation}$ (promoting target-task learning) and $\textbf{stability}$ (preserving pre-trained capabilities), and makes each objective $\textbf{path-aware}$ by combining the direct-path signal from local token gradients with the downstream causal-path signal in autoregressive generation. Since retention data are typically unavailable, AlphaToken approximates stability via a $\textbf{Fisher-drift proxy}$ anchored at the pre-trained reference model. For efficient computation, we extend Ghost Dot-Product to token-level valuation. AlphaToken masks low-value response tokens during fine-tuning and preference optimization, concentrating training signals on more valuable positions. Experiments show that AlphaToken improves post-training performance and mitigates catastrophic forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。