通过对比解码增强模型对输入标签映射的理解，提升少样本学习效果。
Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding
- 用正负例对比方式强化模型对输入标签对应关系的关注。
- 在7个NLU任务上平均提升1.8分，最大提升达1.8点。
- 无需训练，适配多种示例选择方法，通用性强。
大语言模型（LLMs）通过上下文学习（ICL）在多项任务中表现优异，仅需少量任务示例即可引导预测。然而，已有研究指出，模型在ICL中常忽略输入-标签映射信息，更依赖预训练知识。为此，本文提出上下文对比解码（ICCD），通过对比正负示例的输出分布，强调输入-标签映射关系。在7个自然语言理解（NLU）任务上的实验表明，该方法在6种不同规模的LLM上均带来稳定且显著的提升，平均提升达+1.8，最大提升为+1.8，且无需额外训练。该方法具备广泛适用性，可与多种示例选择策略兼容。代码与脚本已开源：https://github.com/Romainpkq/CD_ICL。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at a range of tasks through in-context learning (ICL), where only a few task examples guide their predictions. However, prior research highlights that LLMs often overlook input-label mapping information in ICL, relying more on their pre-trained knowledge. To address this issue, we introduce In-Context Contrastive Decoding (ICCD), a novel method that emphasizes input-label mapping by contrasting the output distributions between positive and negative in-context examples. Experiments on 7 natural language understanding (NLU) tasks show that our ICCD method brings consistent and significant improvement (up to +1.8 improvement on average) upon 6 different scales of LLMs without requiring additional training. Our approach is versatile, enhancing performance with various demonstration selection methods, demonstrating its broad applicability and effectiveness. The code and scripts are released at https://github.com/Romainpkq/CD_ICL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。