Trove让密集检索实验更灵活高效，支持动态数据处理与多节点部署。
Trove: A Flexible Toolkit for Dense Retrieval
- 支持在线加载和动态处理数据集，无需预存多份副本。
- 数据管理降低2.6倍内存占用，推理速度随节点数线性提升。
- 低代码统一流程，适合需要快速迭代的检索研究者。
我们提出Trove，一个易用的开源检索工具包，简化研究实验的同时保持灵活性和速度。首次引入高效的动态数据管理功能，仅需少量代码即可实时加载、过滤、选择、转换和组合检索数据集，使用户能轻松尝试不同数据配置，而无需计算并存储大型数据集的多个副本。Trove高度可定制：除大量内置选项外，支持自由修改或替换组件。其提供低代码统一评估与难负样本挖掘管道，支持多节点执行且无需修改代码。数据管理特性将内存消耗降低2.6倍。此外，其易用的推理流水线无额外开销，推理时间随可用节点数线性下降。最重要的是，我们展示了Trove如何简化检索实验并支持任意自定义，推动探索性研究。
原文摘要 · Abstract (English)
We introduce Trove, an easy-to-use open-source retrieval toolkit that simplifies research experiments without sacrificing flexibility or speed. For the first time, we introduce efficient data management features that load and process (filter, select, transform, and combine) retrieval datasets on the fly, with just a few lines of code. This gives users the flexibility to easily experiment with different dataset configurations without the need to compute and store multiple copies of large datasets. Trove is highly customizable: in addition to many built-in options, it allows users to freely modify existing components or replace them entirely with user-defined objects. It also provides a low-code and unified pipeline for evaluation and hard negative mining, which supports multi-node execution without any code changes. Trove's data management features reduce memory consumption by a factor of 2.6. Moreover, Trove's easy-to-use inference pipeline incurs no overhead, and inference times decrease linearly with the number of available nodes. Most importantly, we demonstrate how Trove simplifies retrieval experiments and allows for arbitrary customizations, thus facilitating exploratory research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。