用细粒度文本反馈引导模型逐步改写，提升指令遵循能力。
Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans
- 通过标记喜欢/不喜欢的文本片段，引导模型逐段改进输出。
- 相比全量重写或简单对比排序，改进链方法使模型更高效学习。
- 适合需要精准控制生成质量的场景，如对话系统优化。
我们提出一种基于反馈驱动改进链的语言模型微调方法与数据集。给定模型输出后，标注者通过标记‘喜欢’和‘不喜欢’的文本片段，并说明原因来提供细粒度反馈。基础模型按从左到右顺序重写不满意的片段，形成一系列增量式改进。从改进链中相邻步骤构建偏好对，用于直接对齐训练，使模型学会从局部、有针对性的修改中学习。实验表明，该方法在指令遵循任务上优于基于标准A/B偏好评分或完整对比重写的直接对齐方法，证明结构化、基于修订的监督能实现更高效、更有效的偏好微调。
原文摘要 · Abstract (English)
We present a method and dataset for fine-tuning language models with preference supervision using feedback-driven improvement chains. Given a model response, an annotator provides fine-grained feedback by marking ``liked'' and ``disliked'' spans and specifying what they liked or disliked about them. The base model then rewrites the disliked spans accordingly, proceeding from left to right, forming a sequence of incremental improvements. We construct preference pairs for direct alignment from each adjacent step in the chain, enabling the model to learn from localized, targeted edits. We find that our approach outperforms direct alignment methods based on standard A/B preference ranking or full contrastive rewrites, demonstrating that structured, revision-based supervision leads to more efficient and effective preference tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。