研究大模型如何逐步学会关注提示中的关键特征，揭示训练阶段对理解能力的影响。
Emergence of Context Characteristics Sensitivity in Large Language Models
- 通过分阶段训练观察模型对提示特征的敏感度变化
- SFT阶段使模型更偏好易懂提示，如长文本、高相似度
- 不同数据集会强化或纠正这种偏好，需平衡数据设计
在指令微调（IFT）过程中，大语言模型通过利用上下文来回答问题。尽管已有研究探讨了上下文特征与模型使用之间的关系，但这些分析多局限于推理阶段，未能揭示这些关系是如何在训练中逐步形成的。本文测量了四个模型在三个数据集上，经历监督微调（SFT）、直接偏好优化（DPO）和基于可验证奖励的强化学习（RLVR）各阶段时，对上下文特征敏感度的变化。结果表明，SFT阶段使模型更倾向于使用易于理解的上下文，如长度较长、与查询相似度较高、表达流畅的文本。而后续阶段的行为可能进一步强化或修正这一偏好，具体取决于训练数据集。研究发现，上下文使用策略在每一阶段都会被主动重塑，因此设计均衡的IFT数据集对确保模型稳健地利用上下文至关重要。
原文摘要 · Abstract (English)
During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query. While prior work has studied how context characteristics correlate with context usage by the LLM, this analysis has been limited to inference time, leaving open how these relationships are acquired in the first place. Here, we measure how models' sensitivity to such characteristics shifts across successive IFT stages: supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning with verifiable rewards (RLVR). Experiments across four models and three datasets show that SFT makes models more likely to use contexts that are easy to understand, such as containing high length, context-query similarity, and fluency. Post-SFT dynamics may either reinforce or resolve these preferences depending on the training dataset. Our findings reveal that context usage is actively reshaped at each IFT stage, and designing a balanced IFT dataset is important in ensuring robust context utilization of instruction-tuned models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。