用上下文示例和集成学习自动构建文本摘要评分函数，兼顾全面性和简洁性。
Unifying Conformal Language Tasks with In-Context Ensembles
- 通过上下文示例和模型集成生成评分函数，减少人工设计。
- 在7个NLP任务上实现覆盖率不变前提下，显著提升内容简洁性。
- 理论分析表明：模型多样性越高，集成效果越优，有明确改进上限。
许多自然语言处理任务，如摘要生成和抽取式问答，本质上是从文档中检索相关内容，在满足覆盖性（保留足够相关信息以达成目标）和简洁性（尽可能剔除无关内容）双重约束下进行。已有方法利用置信性预测（conformal prediction）保证覆盖性，但需通过精心设计的评分函数来优化简洁性。当前最先进的评分函数依赖人工编写的LLM提示词，要求模型评估内容重要性，但该过程劳动密集且任务特定。本文提出Conformal Relevance框架，利用上下文学习中的示例筛选与模型集成，自动构建评分函数，在保持覆盖性的前提下显著提升简洁性，且仅需极少人工干预。我们在7个NLP任务上验证了该框架的有效性，并从理论上研究了集成评分的多样性影响，给出了互补性条件——即何种情况下集成能改善最坏情况下的句子得分，以及集成改进的饱和界限。
原文摘要 · Abstract (English)
Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。