用市场设计方法优化数据定价,让贡献者说真话、市场更高效。
Do Data Valuations Make Good Data Prices?
- 从市场设计视角出发,改进数据支付机制
- 现有方法难保成本真实,导致市场低效
- 引入迈尔森和VCG机制,实现激励相容
随着大语言模型日益依赖外部数据源,如何补偿数据贡献者成为核心问题。本文从市场设计视角出发,将支付视为对数据所有者收集与共享数据所承担异质性私人成本的补偿。我们发现,主流估值方法如留一法和数据沙普利值难以实现有效支付,因其无法保证成本的诚实申报,导致市场效率低下。为此,我们借鉴机制设计中的成熟支付规则——迈尔森与维克里-克拉克-格罗夫斯(VCG)机制,将其适配至数据市场场景。研究表明,迈尔森支付是买家视角下最小且最优的可信机制;同时识别出买卖双方均获满意效用并实现市场效率的条件。研究强调,在数据估值设计中必须考虑激励相容性,为构建更稳健高效的數據市场提供路径。框架可直接应用于现实场景,文中通过模拟基于LLM的检索增强生成(RAG)市场在医学难题问答任务中的贡献者补偿,验证其可行性。
原文摘要 · Abstract (English)
As large language models increasingly rely on external data sources, compensating data contributors has become a central concern. But how should these payments be devised? We revisit data valuations from a $\textit{market-design perspective}$ where payments serve to compensate data owners for the $\textit{private}$ heterogeneous costs they incur for collecting and sharing data. We show that popular valuation methods-such as Leave-One-Out and Data Shapley-make for poor payments. They fail to ensure truthful reporting of the costs, leading to $\textit{inefficient market}$ outcomes. To address this, we adapt well-established payment rules from mechanism design, namely Myerson and Vickrey-Clarke-Groves (VCG), to the data market setting. We show that Myerson payment is the minimal truthful mechanism, optimal from the buyer's perspective. Additionally, we identify a condition under which both data buyers and sellers are utility-satisfied, and the market achieves efficiency. Our findings highlight the importance of incorporating incentive compatibility into data valuation design, paving the way for more robust and efficient data markets. Our data market framework is readily applicable to real-world scenarios. We illustrate this with simulations of contributor compensation in an LLM based retrieval-augmented generation (RAG) marketplace tasked with challenging medical question answering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。