用AI裁判替代人工反馈,让小模型写出更优中文祝福语
Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards
- 用多智能体筛选优质数据构建奖励模型
- 基于原则的AI裁判使生成质量显著提升
- 无需大量人工标注,适合资源有限的研究者
大型语言模型在创意写作上表现优异,但计算开销大。提升小型语言模型(SLMs)是可行替代方案,但当前监督微调难以保证新颖性,人类反馈强化学习成本过高。本文在强化学习从AI反馈(RLAIF)框架下,探索两种不同策略以激发70亿参数小模型生成中文问候语的创造力。第一种采用多智能体拒收采样框架构建的高质量偏好数据训练的奖励模型;第二种创新性地使用基于原则的LLM作为裁判,通过带反思机制的对抗训练优化其奖励函数,直接提供反馈信号。实验表明,两种方法均显著优于基线,而基于原则的AI裁判在生成质量、训练效率和降低对人工标注依赖方面更具优势,展现出更可扩展的有效路径。自动化评估结果与人类判断高度一致。代码与数据已公开。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable creative writing capabilities, yet their substantial computational demands hinder widespread use. Enhancing Small Language Models (SLMs) offers a promising alternative, but current methods like Supervised Fine-Tuning (SFT) struggle with novelty, and Reinforcement Learning from Human Feedback (RLHF) is costly. This paper explores two distinct AI-driven reward strategies within a Reinforcement Learning from AI Feedback (RLAIF) framework to ignite the creative writing of a 7B-parameter SLM, specifically for generating Chinese greetings. The first strategy employs a RM trained on high-quality preference data curated by a novel multi-agent rejection sampling framework designed for creative tasks. The second, more novel strategy utilizes a principle-guided LLM-as-a-Judge, whose reward function is optimized via an adversarial training scheme with a reflection mechanism, to directly provide reward signals. Comprehensive experiments reveal that while both approaches significantly enhance creative output over baselines, the principle-guided LLM-as-a-Judge demonstrably yields superior generation quality. Furthermore, it offers notable advantages in training efficiency and reduced dependency on human-annotated data, presenting a more scalable and effective path towards creative SLMs. Our automated evaluation methods also exhibit strong alignment with human judgments. Our code and data are publicly available at https://github.com/weixiaolong94-hub/Igniting-Creative-Writing-in-Small-Language-Models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。