arXiv:2603.22376cs.IRcs.AI2026-03被引 1

AI助手自动完成搜索排序研究闭环,一周内提升0.201%效果。

Closing the Auto-Research Loop: An AI Co-Scientist for Production Search Ranking

  • 用大模型代理+云算力实现从想法到实验的全流程自动化迭代。
  • 在生产任务中实现0.201%的离线性能提升,相当于额外0.083%的自动增益。
  • 能发现跨领域通用技术(如多阶段学习率),适合算法团队快速试错。

我们提出AI协同科学家框架,为大型在线旅游平台的生产级搜索排序系统闭合研究闭环——通过将大语言模型代理与云端计算直接连接,实现想法生成、代码实现、GPU实验和结果分析的端到端自动迭代,人类科学家全程参与。采用混合代理架构:单模型代理处理常规任务,多模型共识(GPT-5.2、Gemini Pro 3、Claude Opus 4.5)用于高风险决策。在生产排序任务中,人工设计的Transformer基线(V2)相较预Transformer基线(V1)提升+0.118%;在此基础上,AI协同科学家的自动化循环贡献额外+0.083%,合计获得+0.201%的离线增益,仅耗时约一周(单次运行数据;统计显著性另文讨论)。最有效的AI建议包括统一长序列布局、槽位类型嵌入和多阶段学习率调度,这些在自然语言处理与视觉领域已是常规实践,但在本生产系统中缺失,表明大模型代理可作为跨领域知识的桥梁。同时报告部署背景、负面结果及经验教训。

原文摘要 · Abstract (English)

We present an AI Co-Scientist framework that closes the research loop for the production search-ranking system of a large online travel platform -- pairing LLM agents with direct cloud-compute access so that idea generation, code implementation, GPU experimentation, and result analysis iterate end-to-end with a human scientist in the loop. The framework uses a hybrid agent architecture: single-LLM agents handle routine work, while multi-LLM consensus (GPT-5.2, Gemini Pro 3, Claude Opus 4.5) is invoked for higher-stakes decisions. On the production ranking task, a human-designed transformer baseline (V2) yielded $+0.118\%$ over a pre-transformer baseline (V1); the AI Co-Scientist's automated loop on top of V2 contributed an additional $+0.083\%$, for a combined $+0.201\%$ offline gain delivered in roughly one extra week of wall-clock time (single-run numbers; statistical limits discussed in the paper). The most useful AI proposals -- unified long-sequence layouts, slot-type embeddings, and multi-phase learning-rate schedules -- are standard practice in NLP and Vision but were absent from our production stack, suggesting that LLM agents can serve as cross-disciplinary connectors for ranking teams. We also report deployment context, negative results, and lessons learned.

AI科研搜索排序自动化大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。