通过迭代优化提升社交聊天大模型的互动性与可控性。
CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production
- 构建飞轮流程，融合数据筛选、奖励建模与强化学习持续优化。
- 8个新模型中7个显著提升互动度，最高增益达19.4%。
- 适合大规模生产部署的模型迭代系统，关注真实用户反馈。
本文介绍CharacterFlywheel，一个用于在Instagram、WhatsApp和Messenger等生产级社交聊天应用中迭代优化大语言模型（LLMs）的飞轮机制。从LLaMA 3.1出发，我们基于内外部真实用户流量数据，经过15代模型迭代优化。2024年7月至2025年4月期间，通过7天受控A/B测试发现，8个新部署模型中有7个相比基线实现正向提升，最强模型在互动广度上提升8.8%，深度提升19.4%。同时，可控性显著增强：指令遵循率从59.2%提升至84.8%，指令违规率从26.6%降至5.8%。该流程整合数据筛选、奖励建模以估算并插值互动指标、监督微调（SFT）、强化学习（RL），以及离线与在线评估，确保每一步优化可靠推进。还讨论了防止过拟合及应对大规模生产动态的方法。本工作推动了社交类大模型在千万级用户场景下的科学化演进。
原文摘要 · Abstract (English)
This report presents CharacterFlywheel, an iterative flywheel process for improving large language models (LLMs) in production social chat applications across Instagram, WhatsApp, and Messenger. Starting from LLaMA 3.1, we refined models across 15 generations using data from both internal and external real-user traffic. Through continuous deployments from July 2024 to April 2025, we conducted controlled 7-day A/B tests showing consistent engagement improvements: 7 of 8 newly deployed models demonstrated positive lift over the baseline, with the strongest performers achieving up to 8.8% improvement in engagement breadth and 19.4% in engagement depth. We also observed substantial gains in steerability, with instruction following increasing from 59.2% to 84.8% and instruction violations decreasing from 26.6% to 5.8%. We detail the CharacterFlywheel process which integrates data curation, reward modeling to estimate and interpolate the landscape of engagement metrics, supervised fine-tuning (SFT), reinforcement learning (RL), and both offline and online evaluation to ensure reliable progress at each optimization step. We also discuss our methods for overfitting prevention and navigating production dynamics at scale. These contributions advance the scientific rigor and understanding of LLMs in social applications serving millions of users.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。