AI助手提升客服效率与满意度，但高手用反而效果下降。
Generative AI in Action: Field Experimental Evidence from Alibaba's Customer Service Operations
- AI仅在开场提供诊断建议，由人工决定是否采纳。
- 平均提升响应速度和客户评分，但不影响重诉率。
- 低水平客服受益最大，高手使用后质量反而下降。
在与阿里巴巴合作的大规模实地实验中，我们研究了生成式AI助手对电商售后客服表现的影响。数字客服在聊天支持中被随机分配访问生成式AI助手，该助手仅在对话初始阶段生成问题诊断与解决方案建议，人工可选择采纳、修改或忽略。由于保留了人工决策权，我们能够评估访问和使用两种效应。结果显示，生成式AI平均提升了服务速度和主观服务质量（以客户评分衡量），但对客观服务质量（以客户重诉率衡量）无显著影响。这些改进不仅源于自动化，还重塑了人机交互：受试客服响应更快、更主动，客户输入减少，且该趋势延续至后续阶段。然而，平均效应掩盖了个体差异：低绩效客服获益最多，表明生成式AI有助于缩小能力差距；而高绩效客服则出现主观与客观服务质量双重下降，表现为使用后离线时间增加、回复延迟、即时重诉率上升，暗示其工作流程被干扰。因此，成功的生成式AI部署需根据代理技能进行分层评估与渐进推广。
原文摘要 · Abstract (English)
In collaboration with Alibaba, we study how a generative AI assistant affects service performance in e-commerce after-sales operations. In a large-scale field experiment, human agents providing digital chat support were randomly assigned access to a gen AI assistant. The assistant drafts issue diagnoses and solution proposals in the opening stage only; agents can adopt, modify, or disregard them. Because of this discretion, we estimate the effects of both gen AI access and usage. On average, gen AI improves service speed and subjective service quality, measured by customer ratings, but has no significant effect on objective service quality, measured by customer retrials. These gains come from more than automation. Gen AI reshapes agent-customer interactions: treated agents respond faster and take a more proactive role, while customers provide less input; both patterns persist into later chat stages. These average effects, however, mask heterogeneity across agents. Lower-performing agents benefit the most, indicating that gen AI can narrow performance gaps. Top-performing agents experience declines in both subjective and objective service quality. This decline is consistent with workflow disruption: among top performers, gen AI use increases shift-away time, response delays, and immediate customer retrials, suggesting weakened service continuity in the focal chat. Successful gen AI deployment therefore requires careful performance evaluation and rollout tailored to agent skill.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。