提出ICPC方法，让大模型更快处理长提示。
ICPC: In-context Prompt Compression with Faster Inference
- 用编码器计算词出现概率，结合信息函数筛选关键词
- 压缩后信息损失小，推理速度提升显著
- 适合需要高效处理长文本的NLP场景
尽管大型语言模型（LLMs）取得了成功，但其固定输入长度限制了长提示的使用。为此，提示压缩成为有前景的解决方案，通过去除冗余词来缩短提示。然而，现有方法需额外计算资源并带来内存开销。本文提出ICPC（In-context Prompt Compression），一种新颖且可扩展的提示压缩方法，能自适应地减少提示长度。核心思想是利用编码器计算每个词在提示中出现的概率，并通过信息函数衡量每个词携带的信息量，从而有效降低压缩过程中的信息损失，提升压缩速度。实验表明，ICPC能有效压缩不同类别长文本，在多种自然语言处理任务上实现更优性能与更快推理速度。
原文摘要 · Abstract (English)
Despite the recent success of Large Language Models (LLMs), it remains challenging to feed LLMs with long prompts due to the fixed size of LLM inputs. As a remedy, prompt compression becomes a promising solution by removing redundant tokens in the prompt. However, using LLM in the existing works requires additional computation resources and leads to memory overheads. To address it, we propose ICPC (In-context Prompt Compression), a novel and scalable prompt compression method that adaptively reduces the prompt length. The key idea of ICPC is to calculate the probability of each word appearing in the prompt using encoders and calculate information carried by each word through the information function, which effectively reduces the information loss during prompt compression and increases the speed of compression. Empirically, we demonstrate that ICPC can effectively compress long texts of different categories and thus achieve better performance and speed on different types of NLP tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。