用真人评分数据和偏好优化,让机器生成更符合人类审美的改写类型。
Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data
- 用真人打分数据直接优化模型偏好,替代传统自动指标。
- 改写准确率提升3个百分点,人类偏好度提高7个百分点。
- 适合想提升文本生成质量的NLP研究者与应用开发者。
改写旨在重述语义以增强文本简化、机器翻译和问答等应用效果。特定改写类型有助于精准语义分析和鲁棒语言模型构建。然而,现有方法因依赖自动化指标和有限的人工标注数据,常与人类偏好不符,忽视语义保真度和语言转换的关键特征。本研究通过使用真人评分的改写类型数据集,并引入直接偏好优化(DPO)对模型进行训练,使其输出更贴近人类判断。基于DPO的训练使改写类型生成准确率比监督基线提升3个百分点,人类偏好评分提高7个百分点。研究还构建了一个新的真人标注数据集,支持未来更严谨的评估。此外,改写类型检测模型在添加/删除、同极性替换、标点变化上的F1得分分别为0.91、0.78和0.70。结果表明,偏好数据与DPO训练可生成更可靠、语义准确的改写内容,推动摘要生成和问答系统改进。该检测模型超越自动指标,为改写质量评估提供更可靠的框架,助力面向用户需求的语言生成研究,奠定以人为中心的评估基础。
原文摘要 · Abstract (English)
Paraphrasing re-expresses meaning to enhance applications like text simplification, machine translation, and question-answering. Specific paraphrase types facilitate accurate semantic analysis and robust language models. However, existing paraphrase-type generation methods often misalign with human preferences due to reliance on automated metrics and limited human-annotated training data, obscuring crucial aspects of semantic fidelity and linguistic transformations. This study addresses this gap by leveraging a human-ranked paraphrase-type dataset and integrating Direct Preference Optimization (DPO) to align model outputs directly with human judgments. DPO-based training increases paraphrase-type generation accuracy by 3 percentage points over a supervised baseline and raises human preference ratings by 7 percentage points. A newly created human-annotated dataset supports more rigorous future evaluations. Additionally, a paraphrase-type detection model achieves F1 scores of 0.91 for addition/deletion, 0.78 for same polarity substitution, and 0.70 for punctuation changes. These findings demonstrate that preference data and DPO training produce more reliable, semantically accurate paraphrases, enabling downstream applications such as improved summarization and more robust question-answering. The PTD model surpasses automated metrics and provides a more reliable framework for evaluating paraphrase quality, advancing paraphrase-type research toward richer, user-aligned language generation and establishing a stronger foundation for future evaluations grounded in human-centric criteria.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。