用专家反馈优化代码助手的检索策略，提升查准率。
FLAIR: Feedback Learning for Adaptive Information Retrieval
- 通过用户反馈和文档合成问题收集领域指标，离线存储
- 在线阶段结合相似度与反馈指标，双轨排序提升检索效果
- 已在微软千人级系统落地，对新旧查询均有效
大型语言模型的发展推动了协作式智能助手在复杂技术场景中的应用，凸显了专用信息检索方案的需求。本文提出FLAIR，一种轻量级反馈学习框架，通过整合领域专家反馈来动态调整协作式系统的检索策略。该框架分两阶段运行：离线阶段从用户反馈和文档合成问题中获取指标，并以去中心化方式存储；在线阶段采用双轨排序机制，融合原始相似度分数与采集到的指标。这种迭代设计可持续优化任意查询的检索性能。对FLAIR的广泛真实世界评估显示，其在已见和未见查询上均显著优于现有方法。该系统已成功集成至微软的Copilot DECO，服务数千名用户，验证了其在实际环境中的可扩展性与有效性。
原文摘要 · Abstract (English)
Recent advances in Large Language Models (LLMs) have driven the adoption of copilots in complex technical scenarios, underscoring the growing need for specialized information retrieval solutions. In this paper, we introduce FLAIR, a lightweight, feedback learning framework that adapts copilot systems' retrieval strategies by integrating domain-specific expert feedback. FLAIR operates in two stages: an offline phase obtains indicators from (1) user feedback and (2) questions synthesized from documentation, storing these indicators in a decentralized manner. An online phase then employs a two-track ranking mechanism to combine raw similarity scores with the collected indicators. This iterative setup refines retrieval performance for any query. Extensive real-world evaluations of FLAIR demonstrate significant performance gains on both previously seen and unseen queries, surpassing state-of-the-art approaches. The system has been successfully integrated into Copilot DECO, serving thousands of users at Microsoft, demonstrating its scalability and effectiveness in operational environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。