PILOT让推荐系统实验变主动,自动设计个性化策略并积累经验。
PILOT Technical Report

- 构建三角色代理框架,实现从实验设计到复盘的全流程自动化。
- 在淘宝平台提升核心指标最高达1.60%,搜索效率提高40个百分点。
- 适合需要持续优化推荐系统的工业级团队,尤其关注可复用方法论。
现有推荐系统优化的智能体方法仍以被动响应为主:仅根据观测指标变化调整参数,缺乏主动设计可控实验、用户分群个性化策略及跨任务方法积累的能力。本文提出PILOT(Proactive Insight Learner for Online Tree-Experiments),一个基于LLM的代理框架,在受限控制回路中组织三种角色:(1)实验管理器负责全生命周期管理——任务接入、观察治理、异常恢复与事后分析,仅从规则生成的合法指令包中选择;(2)搜索规划器在管理器请求时提出用户分群级决策树方案;(3)记忆策展者异步将实验结果提炼为策略级知识与可追溯的方法论,且与主流程隔离失败影响。管理器使代理具备主动性,规划器实现群体级个性化,策展者将每次任务转化为下一次的学习资源。在淘宝平台部署5个实验桶,对比无生命周期治理与结构化假设测试的自由探索代理ROAM,PILOT在指标上最高提升1.40% IPV、1.60% Core IPV、0.96%交易量、1.50%交易额,优于ROAM最佳表现(+1.00% IPV、+0.90% Core IPV、+0.60%交易量、+1.13%交易额),同时搜索效率从53.3%提升至93.3%(+40个百分点),整个实验周期无需人工干预。
原文摘要 · Abstract (English)
Existing agentic approaches for recommendation system optimization remain fundamentally reactive: they adjust parameters in response to observed metric changes but lack the ability to proactively design controlled experiments, personalize strategies at the user-segment level, or accumulate reusable experimental methodology across tasks. We present PILOT (Proactive Insight Learner for Online Tree-Experiments), an LLM-agent framework that organizes three roles within a constrained control loop where deterministic services enforce all safety, statistical, and permission boundaries: (1) an Experiment Manager that drives the full experiment lifecycle -- task intake, observation governance, anomaly recovery, and postmortem -- by selecting only from a rule-generated legal-command envelope; (2) a Search Planner that proposes candidate decision trees for user-segment-level personalization, invoked only when the Manager requests planning; and (3) a Memory Curator that asynchronously distills experiment outcomes into strategy-level domain knowledge and provenance-tracked methodology, failure-isolated from the main loop. The Manager makes the agent proactive, the Planner enables population-level personalization beyond global tuning, and the Curator turns every completed task into a learning opportunity for the next. Deployed on Taobao's platform with 5 experimental buckets, PILOT is compared against ROAM(Reactive Optimization with Agent-driven Moves), a free-exploration agent without lifecycle governance or structured hypothesis testing. PILOT achieves up to +1.40% IPV, +1.60% Core IPV, +0.96% transaction count, and +1.50% transaction amount, improving over ROAM's best results (+1.00% IPV, +0.90% Core IPV, +0.60% transaction count, +1.13% transaction amount) while raising search efficiency from 53.3% to 93.3% (+40 pp), with no human intervention throughout the experimental cycle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。