解决大模型对标签长度的偏差问题，提升多标签预测准确性
Mitigating Label Length Bias in Large Language Models
- 在完整标签层面进行归一化与校准，缓解标签长度带来的偏差
- 在多个数据集上实现最高10%的F1提升，效果显著优于旧方法
- 适合需要可靠置信度和少样本学习的现实应用场景
大型语言模型（LLMs）虽具备强大的零样本和少样本学习能力，但在预测候选选项时存在标签偏差问题，现有校准方法未考虑多标记类别标签引发的偏差。本文提出一种名为归一化上下文校准（NCC）的新方法，通过在全标签层级进行归一化与校准，有效缓解标签长度偏差。NCC在多个数据集和模型上均取得统计显著提升，最大F1提升达10%。此外，该方法可拓展至多项选择题问答等任务。分析表明，结合上下文学习后，NCC对少样本示例选择更不敏感，所需示例更少即可达到良好性能，并生成更可靠的置信度估计。这些发现凸显了消除全标签偏差对提升基于LLM方法性能与鲁棒性的重要性，尤其在真实场景中类别标签天然为多标记的情况下。
原文摘要 · Abstract (English)
Large language models (LLMs) are powerful zero- and few-shot learners. However, when predicting over a set of candidate options, LLMs suffer from label biases, and existing calibration methods overlook biases arising from multi-token class labels. We tackle an issue we call label length bias, where labels of different lengths are treated inconsistently, even after standard length normalization. To mitigate it, we propose normalized contextual calibration (NCC), an effective method that normalizes and calibrates predictions at the full-label level. NCC achieves statistically significant improvements over prior approaches across multiple datasets and models, with gains of up to 10% F1. Moreover, NCC extends bias mitigation to broader tasks such as multiple-choice question answering. Our analysis shows that, when combined with in-context learning, NCC is less sensitive to few-shot example selection, requires fewer examples for competitive performance, and produces more reliable confidence estimates. These findings highlight the importance of mitigating full-label biases to improve the performance and robustness of LLM-based methods, particularly in real-world applications where class labels naturally consist of multiple tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。