用可调节的分块策略提升大模型少样本学习效果
Submodular Context Partitioning and Compression for In-Context Learning
- 基于子模函数设计分块选择机制，平衡信息多样性与局部一致性
- 在多个数据集上提升不同规模模型的少样本学习性能
- 适合需要高效处理大量示例的少样本学习场景
上下文学习（ICL）使大型语言模型在无需训练的情况下实现高效少样本学习，但受变压器模型二次输入复杂度限制，最大示例数受限。现有高效ICL方法虽通过分块处理（如集成、压缩、交叉注意力）提升效率，却常忽略不同分块策略带来的信息冗余或表征不足问题，导致性能不佳。为此，我们提出Sub-CP，一种块感知的上下文选择框架，利用子模目标控制块间多样性。Sub-CP支持灵活的选取策略，使每块可在全局多样与局部一致之间自由调节，兼顾语义结构精细控制与预计算可行性。在多个数据集上的多样化任务实验表明，Sub-CP在不同模型规模下均显著提升性能。
原文摘要 · Abstract (English)
In-context learning (ICL) enables efficient few-shot learning in large language models (LLMs) without training, but suffers from the quadratic input complexity of transformers, limiting the maximum number of exemplars. While various efficient ICL approaches partition the context into blocks to process (e.g., ensembling, compression, cross-attention), they often ignore the information redundancy or under-representation caused by different partition strategies, leading to suboptimal performance. To tackle this problem, we propose Sub-CP, a block-aware context selection framework that leverages submodular objectives to control block diversity. Sub-CP supports a flexible spectrum of selection strategies, allowing each block to range from globally diverse to locally coherent. This allows fine-grained control over semantic structure while enabling precomputation. Extensive experiments across diverse tasks on multiple datasets show that Sub-CP consistently improves performance across model scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。