arXiv:2507.21939cs.IR2025-07被引 1

将高维索引转为文件结构,提升可读性与兼容性。

The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS

  • 把eCP索引映射为文件系统,用标准文件库读写
  • 内存受限时内存占用极低,适合多索引共存环境
  • 虽查询慢但代码简洁,便于调试与分析

现代分析流水线常部署多个依赖近似最近邻(ANN)索引的深度学习与检索模型。尽管主流ANN索引如HNSW和IVF为内存型,但多个索引竞争有限的GPU/CPU内存,迫使采用磁盘型索引(如DiskANN或eCP)。传统实现中,索引以复杂数据结构序列化至磁盘,内存型索引启动时全载入,磁盘型则按需读取。可视化或分析索引需复杂编码,或复现解析逻辑。本文提出eCP-FS,将eCP索引映射为文件结构,利用文件库实现跨语言、甚至人类可读。缺点是序列化后体积大,搜索开销高。实验表明,虽然eCP-FS查询较慢,但在无内存约束下仍具竞争力;在内存受限场景中,其极小内存占用使其成为资源受限或多索引环境的理想选择。

原文摘要 · Abstract (English)

Modern analytical pipelines routinely deploy multiple deep learning and retrieval models that rely on approximate nearest-neighbor (ANN) indexes to support efficient similarity-based search. While many state-of-the-art ANN-indexes are memory-based (e.g., HNSW and IVF), using multiple ANN indexes creates a competition for limited GPU/CPU memory resources, which in turn necessitates disk-based index structures (e.g., DiskANN or eCP). In typical index implementations, the main component is a complex data structure that is serialized to disk and is read either fully at startup time, for memory-based indexes, or incrementally at query time, for disk-based indexes. To visualize the index structure, or analyze its quality, complex coding is needed that is either embedded in the index implementation or replicates the code that reads the data structure. In this paper, we consider an alternative approach that maps the data structure to a file structure, using a file library, making the index easily readable for any programming language and even human-readable. The disadvantage is that the serialized index is verbose, leading to overhead of searching through the index. The question addressed in this paper is how severe this performance penalty is. To that end, this paper presents eCP-FS, a file-based implementation of eCP, a well-known disk-based ANN index. A comparison with state-of-the-art indexes shows that while eCP-FS is slower, the implementation is nevertheless somewhat competitive even when memory is not constrained. In a memory-constrained scenario, eCP-FS offers a minimal memory footprint, making it ideal for resource-constrained or multi-index environments.

索引结构文件系统ANN

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。