用子模互信息提升上下文学习中示例的质量与多样性
InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity
- 基于子模互信息设计统一筛选策略，兼顾示例相关性与多样性
- 在九个基准数据集上显著提升上下文学习性能，效果优于现有方法
- 适合需要高效检索与高质量示例的问答系统研究者
本文提出InSQuAD，通过子模互信息（SMI）增强上下文学习（ICL）模型中示例的质量与多样性。该方法将ICL任务建模为有针对性的示例选择问题，提出基于SMI的统一筛选策略，挖掘既相关又多样化的上下文示例。针对现有检索模型过度关注查询相关性而忽视多样性的缺陷，InSQuAD引入组合式训练范式，通过新型似然损失学习SMI函数参数，以显式强化检索结果的质量与多样性。为进一步辅助训练，我们对现有的多跳问答数据集进行了扩充，加入合成生成的改写语句。结合该训练策略与新的针对性选择方法，在九个基准数据集上的实验表明，该方法显著提升了ICL性能，验证了其有效性。
原文摘要 · Abstract (English)
In this paper, we introduce InSQuAD, designed to enhance the performance of In-Context Learning (ICL) models through Submodular Mutual Information} (SMI) enforcing Quality and Diversity among in-context exemplars. InSQuAD achieves this through two principal strategies: First, we model the ICL task as a targeted selection problem and introduce a unified selection strategy based on SMIs which mines relevant yet diverse in-context examples encapsulating the notions of quality and diversity. Secondly, we address a common pitfall in existing retrieval models which model query relevance, often overlooking diversity, critical for ICL. InSQuAD introduces a combinatorial training paradigm which learns the parameters of an SMI function to enforce both quality and diversity in the retrieval model through a novel likelihood-based loss. To further aid the learning process we augment an existing multi-hop question answering dataset with synthetically generated paraphrases. Adopting the retrieval model trained using this strategy alongside the novel targeted selection formulation for ICL on nine benchmark datasets shows significant improvements validating the efficacy of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。