用索引加速搜索引擎跨段连接,查询速度提升8倍
Join Indices for Search Engines: a Prunable Parallel Semijoin over Lucene Segments
- 为Lucene段创建可裁剪的并行半连接索引,避免运行时键值转换
- 100万商品连1000万SKU,平均延迟从1934.6ms降至359.8ms
- 适合高并发、多段关联查询场景,尤其适合电商和推荐系统
在搜索引擎中,连接操作长期受限于性能或能力。本文将关系型数据库中的连接索引技术引入基于LSM风格段存储的Lucene系统:对每对父段与子段,构建一个仅追加的序号到序号的连接索引列,避免运行时外部可变长度键的转换。在此基础上,设计一种按父段并行执行的半连接算法,无需全局同步;通过三重裁剪机制(段级、预判最大最小值、文档级两阶段确认及懒加载的半读联合)实现高效剪枝,确保与任意查询组合时不会浪费计算资源。原型作为Apache Solr查询解析器实现,在100万商品与1000万SKU的连接任务上,平均查询延迟降低至原来的1/5.4(359.8ms vs. 1934.6ms),并发度达8时优势进一步扩大至8.3倍;在4个vCPU下,基线吞吐量峰值为1.18查询/秒,而该方法仍持续增长,最高达8.04查询/秒,是基线最佳表现的6.8倍。
原文摘要 · Abstract (English)
Joins are second-class citizens in search engines: existing query-time join implementations in Lucene are limited either in performance or in capability, forcing a choice between fast joins scoped to a single index and slower joins that span independently managed indices. We carry Valduriez's join-index technique from relational systems to Lucene's flush-based (LSM-style) segment storage: for every pair of a parent and a child segment we materialize an append-only, ordinal-to-ordinal join-index column J[c]=p, avoiding any query-time translation of external variable-length keys. On top of this structure we build a semijoin algorithm that is computed per parent segment, in parallel, without a global barrier between stages; it prunes at three levels (segment-level, the first of which comes free from per-segment execution; a-priori min/max; and document-level two-phase confirmation with a lazily accumulated half-read union) so that it composes with arbitrary engine queries instead of wasting computation on matches that a sibling filter would later discard. A prototype implemented as an Apache Solr query parser, benchmarked on 1M products joined against 10M skus, cuts average query latency 5.4 times (359.8,ms vs. 1934.6,ms) relative to Solr's built-in query-time join, and the advantage widens monotonically with load, reaching 8.3 times at a concurrency of eight: on 4 vCPUs the baseline peaks at 1.18 queries/s and then loses throughput, while the join index is still gaining, at 8.04 - 6.8times the baseline's best.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。