用超体积优化平衡摘要的多目标得分，让大模型生成更全面的摘要。
Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization
- 通过超体积方法动态调整奖励，引导模型逼近帕累托最优前沿。
- 在多个数据集上综合得分超越GRPO，各维度表现更均衡。
- 70亿参数模型性能接近GPT-4，且生成更短，适合实际部署。
文本摘要是一项关键任务，需同时优化一致性、连贯性、相关性和流畅性等多个目标，挑战巨大。尽管大语言模型（LLMs）在强化学习（RL）增强下表现出色，但针对基于LLM的摘要多目标优化研究仍较少。本文提出超体积优化（HVO），一种新策略：在强化学习的奖励过程中，利用超体积方法动态调整不同目标组的得分，引导模型逐步逼近帕累托前沿，从而生成多目标平衡的摘要。在多个代表性摘要数据集上的实验表明，该方法在整体得分上优于组相对策略优化（GRPO），且各维度表现更均衡。此外，一个70亿参数的基础模型经HVO增强后，在摘要任务中表现可媲美GPT-4，同时生成长度更短。代码已公开于 https://github.com/ai4business-LiAuto/HVO.git。
原文摘要 · Abstract (English)
Text summarization is a crucial task that requires the simultaneous optimization of multiple objectives, including consistency, coherence, relevance, and fluency, which presents considerable challenges. Although large language models (LLMs) have demonstrated remarkable performance, enhanced by reinforcement learning (RL), few studies have focused on optimizing the multi-objective problem of summarization through RL based on LLMs. In this paper, we introduce hypervolume optimization (HVO), a novel optimization strategy that dynamically adjusts the scores between groups during the reward process in RL by using the hypervolume method. This method guides the model's optimization to progressively approximate the pareto front, thereby generating balanced summaries across multiple objectives. Experimental results on several representative summarization datasets demonstrate that our method outperforms group relative policy optimization (GRPO) in overall scores and shows more balanced performance across different dimensions. Moreover, a 7B foundation model enhanced by HVO performs comparably to GPT-4 in the summarization task, while maintaining a shorter generation length. Our code is publicly available at https://github.com/ai4business-LiAuto/HVO.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。