用小模型压缩提示，让大模型翻译评估更省资源
PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics
- 用微调小模型压缩原始提示，减少大模型输入量
- 在不损失评估质量的前提下，令牌使用量降低2.37倍
- 适合需要高效使用大模型进行翻译评估的团队
自然语言生成内容的质量评估是自然语言处理中的挑战性任务。近期，GPT-4等大型语言模型被用于此目的，但因其复杂的评估提示需大量令牌而计算成本高昂。本文提出一种提示优化方法：利用一个小型微调语言模型压缩输入数据，从而减少下游大模型评估时的令牌使用和计算开销。该方法采用两阶段微调：监督微调后接偏好优化，使模型输出更符合人类偏好。聚焦机器翻译评估，以GEMBA-MQM为基准。结果表明，令牌使用量减少2.37倍，且评估质量未下降。本工作使GEMBA-MQM等先进基于大模型的评估指标更高效、更具成本效益，提升其可及性。
原文摘要 · Abstract (English)
Evaluating the quality of machine-generated natural language content is a challenging task in Natural Language Processing (NLP). Recently, large language models (LLMs) like GPT-4 have been employed for this purpose, but they are computationally expensive due to the extensive token usage required by complex evaluation prompts. In this paper, we propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. Our method involves a two-stage fine-tuning process: supervised fine-tuning followed by preference optimization to refine the model's outputs based on human preferences. We focus on Machine Translation (MT) evaluation and utilize the GEMBA-MQM metric as a starting point. Our results show a $2.37\times$ reduction in token usage without any loss in evaluation quality. This work makes state-of-the-art LLM-based metrics like GEMBA-MQM more cost-effective and efficient, enhancing their accessibility for broader use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。