让大模型回答更精准，按句子给奖励
ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning
- 按句子动态打分，而非整段评价
- 在多个多模态数据集上提升准确率
- 无需额外模型，适合需要精细对齐的场景
直接偏好优化（DPO）因其简洁性和计算效率，在对齐大型语言模型（LLMs）方面受到广泛关注。近期研究已将DPO扩展至多模态场景，取得良好效果。然而，传统DPO依赖二元偏好优化，仅对完整响应进行奖励或惩罚，忽视了细粒度片段的正确性，导致优化效果不佳。根本原因在于优化过程中缺乏细粒度监督。为此，我们提出自适应句级偏好优化（ASPO），通过评估每个句子来实现更精确的偏好优化。基于模型预测动态计算句级自适应奖励，提升响应内容评估能力，且不引入额外模型或参数。实验证明，ASPO显著提升了多模态模型的整体性能，在VQA、Image Captioning等任务上表现优越。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) has gained significant attention for its simplicity and computational efficiency in aligning large language models (LLMs). Recent advancements have extended DPO to multimodal scenarios, achieving strong performance. However, traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness, leading to suboptimal solutions. The root of this issue lies in the absence of fine-grained supervision during the optimization process. To address this, we propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization. By dynamically calculating adaptive rewards at the sentence level based on model predictions, ASPO enhances response content assessment without additional models or parameters. This significantly improves the alignment of multimodal features. Extensive experiments show that ASPO substantially enhances the overall performance of multimodal models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。