通过精简冗余词+小模型重建，让大模型生成更省算力。
TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation
- 大模型生成时主动剔除可推断的无意义词，减少输出长度。
- 在NaLDA数据集上平均节省19.4%的token，GPT-4o效果几乎不变。
- 适合追求高效推理的长文本生成场景，如客服、报告生成。
大型语言模型（LLMs）的高推理成本对长输出任务构成挑战。然而自然语言常含冗余，为优化提供可能。我们发现，通过适当提示，LLM能生成保留核心语义的精炼输出。为此提出TRIM：在推理中，大模型根据上下文剔除预定义的语义无关且易推断词汇；随后由训练好的小型低耗模型重建为完整答案。实验表明，在专为重建任务设计的NaLDA数据集上，该方法使GPT-4o平均节省19.4%的token，评估指标仅轻微下降，证明其在效率与准确性间有效平衡。
原文摘要 · Abstract (English)
The high inference cost of Large Language Models (LLMs) poses challenges, especially for tasks requiring lengthy outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language (i.e., concise outputs that retain essential meaning) when prompted appropriately. We propose TRIM, a pipeline for saving computational cost in which the LLM omits a predefined set of semantically irrelevant and easily inferable words based on the context during inference. Then, a specifically trained smaller language model with lower inference cost reconstructs the distilled answer into the ideal answer. Our experiments show promising results, particularly on the proposed NaLDA evaluation dataset focused on the reconstruction task, with 19.4% saved tokens on average for GPT-4o and only a tiny decrease in evaluation metrics. This suggests that the approach can effectively balance efficiency and accuracy in language processing tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。