arXiv:2502.10151cs.IRcs.DC2025-02被引 2

用大模型生成的语义树实现去中心化搜索,显著提升检索效率与准确率。

Semantica: Decentralized Search using a LLM-Guided Semantic Tree Overlay

  • 基于大模型嵌入构建语义树,用户按文档相似性直接连接。
  • 相比现有方法多发现10倍语义相似用户,相同负载下检索相关文档超2倍。
  • 支持用户语义多样性,适合去中心化系统研究者使用。

中心化搜索引擎虽主导互联网,却导致权力集中。去中心化方案在文档检索精度和速度上难以匹敌。然而,当文档语义被有效捕捉时,语义覆盖网络可接近中心化系统的性能。本文提出一种名为 Semantica 的算法,利用大语言模型生成的嵌入表示构建前缀树(trie),使具有相似文档的用户基于语义直接连接。这提升了用户搜索由其直接或邻近连接者响应的概率。该算法还通过在树中创建‘克隆’用户标识,适应个体用户的语义多样性。实验采用真实工作负载模拟,结果表明:Semantica 比当前最优方法多发现最多10倍的语义相似用户;在相同网络负载下,可检索超过2倍的相关文档。代码已公开,以促进该领域研究。

原文摘要 · Abstract (English)

Centralized search engines are key for the Internet, but lead to undesirable concentration of power. Decentralized alternatives fail to offer equal document retrieval accuracy and speed. Nevertheless, Semantic Overlay Networks can come close to the performance of centralized solutions when the semantics of documents are properly captured. This work uses embeddings from Large Language Models to capture semantics and fulfill the promise of Semantic Overlay Networks. Our proposed algorithm, called Semantica, constructs a prefix tree (trie) utilizing document embeddings calculated by a language model. Users connect to each other based on the embeddings of their documents, ensuring that semantically similar users are directly linked. Thereby, this construction makes it more likely for user searches to be answered by the users that they are directly connected to, or by the users they are close to in the network connection graph. The implementation of our algorithm also accommodates the semantic diversity of individual users by spawning "clone" user identifiers in the tree. Our experiments use emulation with a real-world workload to show Semantica's ability to identify and connect to similar users quickly. Semantica finds up to ten times more semantically similar users than current state-of-the-art approaches. At the same time, Semantica can retrieve more than two times the number of relevant documents given the same network load. We also make our code publicly available to facilitate further research in the area.

去中心化语义搜索LLM应用图网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。