对比多种嵌入模型与管道配置，提升AI搜索准确率。
Evaluating Embedding Models and Pipeline Optimization for AI Search Quality
- 测试不同维度嵌入模型与分块策略对搜索效果的影响
- 高维嵌入（如Qwen3-Embedding-8B/4096）使Top-3准确率达0.571
- 神经重排序器可进一步提升排名精度，适合工程优化参考
我们评估了多种文本嵌入模型及系统管道配置在AI驱动搜索中的表现。对比了sentence-transformer与生成式嵌入模型（如All-MPNet、BGE、GTE、Qwen）在不同维度、索引方法（Milvus HNSW/IVF）和分块策略下的性能。基于美国城市议会会议记录，使用本地大语言模型构建了包含11,975个查询-片段配对的自定义评估数据集。数据流水线包括预处理、每段自动生成问题、人工验证及持续集成/部署（CI/CD）整合。采用基于参考答案的指标衡量检索准确率：Top-K Accuracy与归一化折减累积增益（NDCG）。结果表明，更高维度的嵌入显著提升搜索质量（如Qwen3-Embedding-8B/4096在Top-3准确率约0.571，远高于GTE-large/1024的0.412），且神经重排序器（如BGE交叉编码器）可进一步提升至0.527。更细粒度的分块（512字符比2000字符）也提高准确率。本文讨论各因素影响，并提出未来管道自动化与评估方向。
原文摘要 · Abstract (English)
We evaluate the performance of various text embedding models and pipeline configurations for AI-driven search systems. We compare sentence-transformer and generative embedding models (e.g., All-MPNet, BGE, GTE, and Qwen) at different dimensions, indexing methods (Milvus HNSW/IVF), and chunking strategies. A custom evaluation dataset of 11,975 query-chunk pairs was synthesized from US City Council meeting transcripts using a local large language model (LLM). The data pipeline includes preprocessing, automated question generation per chunk, manual validation, and continuous integration/continuous deployment (CI/CD) integration. We measure retrieval accuracy using reference-based metrics: Top-K Accuracy and Normalized Discounted Cumulative Gain (NDCG). Our results demonstrate that higher-dimensional embeddings significantly boost search quality (e.g., Qwen3-Embedding-8B/4096 achieves Top-3 accuracy about 0.571 versus 0.412 for GTE-large/1024), and that neural re-rankers (e.g., a BGE cross-encoder) further improve ranking accuracy (Top-3 up to 0.527). Finer-grained chunking (512 characters versus 2000 characters) also improves accuracy. We discuss the impact of these factors and outline future directions for pipeline automation and evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。