让大模型推理更公平，避免只看单边文本就下结论。
Task Calibration: Calibrating Large Language Models on Inference Tasks
- 通过重构任务引导模型同时关注前提和假设
- 零样本下在13个任务上显著提升性能
- 不依赖训练数据，适合各类自然语言理解场景
大型语言模型（LLMs）在零样本推理任务中表现出色。然而，它们可能因输入文本与输出标签之间的虚假关联而表现不佳，导致推理时过度依赖前提或假设中的单一成分，而非综合分析两者。为解决这一问题，我们提出任务校准（Task Calibration, TC），一种无需训练、仅在推理阶段使用的校准方法。该方法受互信息启发，通过任务重定义恢复模型的推理能力，促使模型基于前提和假设共同判断，减少对单一成分的过度依赖。实验表明，在13个零样本推理任务中，TC均取得显著性能提升。进一步验证显示，TC在少样本设置及多种自然语言理解任务中同样有效。分析还表明，该方法对提示模板不敏感，且可与其他校准方法结合使用。
原文摘要 · Abstract (English)
Large language models (LLMs) have exhibited impressive zero-shot performance on inference tasks. However, LLMs may suffer from spurious correlations between input texts and output labels, which limits LLMs' ability to reason based purely on general language understanding. In other words, LLMs may make predictions primarily based on premise or hypothesis, rather than both components. To address this problem that may lead to unexpected performance degradation, we propose task calibration (TC), a zero-shot and inference-only calibration method inspired by mutual information which recovers LLM performance through task reformulation. TC encourages LLMs to reason based on both premise and hypothesis, while mitigating the models' over-reliance on individual premise or hypothesis for inference. Experimental results show that TC achieves a substantial improvement on 13 inference tasks in the zero-shot setup. We further validate the effectiveness of TC in few-shot setups and various natural language understanding tasks. Further analysis indicates that TC is also robust to prompt templates and has the potential to be integrated with other calibration methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。