用福利函数平衡搜索结果的相关性与多样性,让推荐更全面。
Welfarist Formulations for Diverse Similarity Search
- 基于经济福利函数构建可自适应调节的相关性与多样性权衡机制。
- 在多个数据集上显著提升多样性,同时保持高相关性,优于固定约束方法。
- 兼容现有近似最近邻算法,适合需要个性化推荐的场景。
最近邻搜索(NNS)是数据结构中的基础问题,广泛应用于网络搜索、推荐系统及检索增强生成(RAG)。在这些新场景中,除了返回结果的相关性外,邻居之间的多样性也至关重要。本文提出基于福利函数的正规化框架,以实现跨属性的多样性。该框架源自数学经济学中的福利函数,满足核心多样性(公平性)与相关性(经济效率)公理。聚焦纳什社会福利,我们发现其目标函数能根据查询动态平衡相关性与多样性,而此前的约束方法需固定多样性水平,无法自适应。此外,该框架提供参数化控制,使从业者可根据任务需求灵活调整权衡。我们设计了具有理论保证的高效近似算法,可集成任意标准的近似最近邻(ANN)方法作为子程序,快速找到近似最大化福利目标的邻居。实验表明,该方法实用且显著提升多样性,同时维持高相关性。
原文摘要 · Abstract (English)
Nearest Neighbor Search (NNS) is a fundamental problem in data structures with wide-ranging applications, such as web search, recommendation systems, and, more recently, retrieval-augmented generations (RAG). In such recent applications, in addition to the relevance (similarity) of the returned neighbors, diversity among the neighbors is a central requirement. In this paper, we develop principled welfare-based formulations in NNS for realizing diversity across attributes. Our formulations are based on welfare functions -- from mathematical economics -- that satisfy central diversity (fairness) and relevance (economic efficiency) axioms. With a particular focus on Nash social welfare, we note that our welfare-based formulations provide objective functions that adaptively balance relevance and diversity in a query-dependent manner. Notably, such a balance was not present in the prior constraint-based approach, which forced a fixed level of diversity and optimized for relevance. In addition, our formulation provides a parametric way to control the trade-off between relevance and diversity, providing practitioners with flexibility to tailor search results to task-specific requirements. We develop efficient nearest neighbor algorithms with provable guarantees for the welfare-based objectives. Notably, our algorithm can be applied on top of any standard ANN method (i.e., use standard ANN method as a subroutine) to efficiently find neighbors that approximately maximize our welfare-based objectives. Experimental results demonstrate that our approach is practical and substantially improves diversity while maintaining high relevance of the retrieved neighbors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。