AI助手自动完成搜索排序研究闭环,一周内提升0.201%效果。
Closing the Auto-Research Loop: An AI Co-Scientist for Production Search Ranking
- 用大模型代理+云算力实现从想法到实验的全流程自动化迭代。
- 在生产任务中实现0.201%的离线性能提升,相当于额外0.083%的自动增益。
- 能发现跨领域通用技术(如多阶段学习率),适合算法团队快速试错。
我们提出AI协同科学家框架,为大型在线旅游平台的生产级搜索排序系统闭合研究闭环——通过将大语言模型代理与云端计算直接连接,实现想法生成、代码实现、GPU实验和结果分析的端到端自动迭代,人类科学家全程参与。采用混合代理架构:单模型代理处理常规任务,多模型共识(GPT-5.2、Gemini Pro 3、Claude Opus 4.5)用于高风险决策。在生产排序任务中,人工设计的Transformer基线(V2)相较预Transformer基线(V1)提升+0.118%;在此基础上,AI协同科学家的自动化循环贡献额外+0.083%,合计获得+0.201%的离线增益,仅耗时约一周(单次运行数据;统计显著性另文讨论)。最有效的AI建议包括统一长序列布局、槽位类型嵌入和多阶段学习率调度,这些在自然语言处理与视觉领域已是常规实践,但在本生产系统中缺失,表明大模型代理可作为跨领域知识的桥梁。同时报告部署背景、负面结果及经验教训。
原文摘要 · Abstract (English)
We present an AI Co-Scientist framework that closes the research loop for the production search-ranking system of a large online travel platform -- pairing LLM agents with direct cloud-compute access so that idea generation, code implementation, GPU experimentation, and result analysis iterate end-to-end with a human scientist in the loop. The framework uses a hybrid agent architecture: single-LLM agents handle routine work, while multi-LLM consensus (GPT-5.2, Gemini Pro 3, Claude Opus 4.5) is invoked for higher-stakes decisions. On the production ranking task, a human-designed transformer baseline (V2) yielded $+0.118\%$ over a pre-transformer baseline (V1); the AI Co-Scientist's automated loop on top of V2 contributed an additional $+0.083\%$, for a combined $+0.201\%$ offline gain delivered in roughly one extra week of wall-clock time (single-run numbers; statistical limits discussed in the paper). The most useful AI proposals -- unified long-sequence layouts, slot-type embeddings, and multi-phase learning-rate schedules -- are standard practice in NLP and Vision but were absent from our production stack, suggesting that LLM agents can serve as cross-disciplinary connectors for ranking teams. We also report deployment context, negative results, and lessons learned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。