通过三元组机制提升语言模型指令遵循能力，减少拒答
Similarity-Distance-Magnitude Language Models
- 用相似性-距离-幅度层指导模型生成，优化输出质量
- 拒答率降低，统计效率优于强基线模型
- 适合需要高可靠性生成的场景，如安全敏感应用
我们提出相似性-距离-幅度（SDM）语言模型，通过微调使模型生成更多处于校准良好的高概率区域的内容。该方法利用最终层的SDM激活层进行二分类，判断指令遵循情况。现有预训练的解码器仅变压器语言模型可通过监督微调转换为SDM模型，训练中使用最终层的SDM激活层估计基底变换，并在对比输入编码方案下采用监督的下一个词元损失，同时在线生成额外难例。实验表明，相比强监督基线，该方法显著降低了拒答率（即提升了统计效率）。
原文摘要 · Abstract (English)
We introduce Similarity-Distance-Magnitude (SDM) language models (LMs), which are sequence prediction models fine-tuned to maximize the proportion of generations in the well-calibrated, high-probability region partitioned by a final-layer SDM activation layer used for binary classification of instruction-following. We demonstrate that existing pre-trained decoder-only Transformer LMs can be readily converted into SDM LMs via supervised fine-tuning, using the final-layer SDM activation layer during training to estimate a change-of-base for a supervised next-token loss over a contrastive input encoding scheme, with additional hard negative examples generated online during training. This results in reduced abstentions (i.e., improved statistical efficiency) compared to strong supervised baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。