用特征匹配替代逐词预测，让语言模型生成更符合语义的完整句子。
Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models
- 通过匹配生成结果的特征分布，优化模型整体输出质量。
- 在问答、编程和翻译任务中准确率超监督微调，且验证交叉熵更低。
- 无需额外验证器，适合追求生成质量而非逐词准确的场景。
交叉熵训练虽可提供密集且可扩展的监督信号，但仅优化教师强制下的下一个词预测，而非模型自回归生成时的序列级行为。本文提出一种面向语言模型微调的特征匹配目标，直接针对完成结果分布的序列级统计特性，提供密集的语义反馈，无需任务特定的验证器或偏好模型。为高效优化该目标，我们提出能量基微调（EBFT），利用分块并行采样从嵌套前缀中并发生成多个自回归路径，批量提取这些路径的特征嵌入，并基于嵌入执行在线策略梯度更新。理论上，我们揭示了EBFT与KL正则化特征匹配及能量模型之间的联系。实验表明，在问答、非结构化编程和翻译任务中，EBFT性能媲美强化学习基准（RLVR），优于监督微调（SFT），且验证交叉熵更低。
原文摘要 · Abstract (English)
Cross-entropy (CE) training provides dense and scalable supervision for language models, but it optimizes next-token prediction under teacher forcing rather than sequence-level behavior under model rollouts. We introduce a feature-matching objective for language-model fine-tuning that targets sequence-level statistics of the completion distribution, providing dense semantic feedback without requiring a task-specific verifier or preference model. To optimize this objective efficiently, we propose energy-based fine-tuning (EBFT), which uses strided block-parallel sampling to generate multiple rollouts from nested prefixes concurrently, batches feature extraction over these rollouts, and uses the resulting embeddings to perform an on-policy policy-gradient update. We present a theoretical perspective connecting EBFT to KL-regularized feature-matching and energy-based modeling. Empirically, across Q&A coding, unstructured coding, and translation, EBFT matches RLVR and outperforms SFT on downstream accuracy while achieving a lower validation cross-entropy than both methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。