用Sigmoid头提升语言模型质量评估，解决多正确输出时概率分散问题。
Sigmoid Head for Quality Estimation under Language Ambiguity
- 在预训练模型上加Sigmoid头，允许多个正确输出同时获得高分
- 训练时用启发式避免选中可能的正确替代词，更贴近真实质量
- 无需人工标注数据，适合未知领域场景
语言模型的概率并非可靠的生成质量评估指标，因自然语言存在歧义。当多个输出选项均有效时，模型概率分布会分散到各选项，导致误判为低质量。根源在于：（1）语言模型最终输出使用softmax激活，无法让多个正确选项同时获得高概率；（2）训练数据为单一、one-hot编码的参考文本，暗示每步仅有一个正确答案。为此，我们提出在预训练语言模型之上训练一个质量评估模块，称为Sigmoid Head。该模块为额外的unembedding头，采用sigmoid激活以解决第一个问题。为应对第二个问题，在训练Sigmoid Head的负采样过程中，引入启发式策略避免选取潜在的正确替代词。Sigmoid Head在训练和推理中计算高效，其输出概率相比原softmax头显著更优，且不依赖人工标注的质量数据，因此对域外情况更具鲁棒性。
原文摘要 · Abstract (English)
Language model (LM) probability is not a reliable quality estimator, as natural language is ambiguous. When multiple output options are valid, the model's probability distribution is spread across them, which can misleadingly indicate low output quality. This issue is caused by two reasons: (1) LMs' final output activation is softmax, which does not allow multiple correct options to receive high probabilities simultaneuously and (2) LMs' training data is single, one-hot encoded references, indicating that there is only one correct option at each output step. We propose training a module for Quality Estimation on top of pre-trained LMs to address these limitations. The module, called Sigmoid Head, is an extra unembedding head with sigmoid activation to tackle the first limitation. To tackle the second limitation, during the negative sampling process to train the Sigmoid Head, we use a heuristic to avoid selecting potentially alternative correct tokens. Our Sigmoid Head is computationally efficient during training and inference. The probability from Sigmoid Head is notably better quality signal compared to the original softmax head. As the Sigmoid Head does not rely on human-annotated quality data, it is more robust to out-of-domain settings compared to supervised QE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。