改变字段顺序会破坏检索效果,新方法让模型无视顺序也能准确查找数据。
Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval

- 用随机打乱字段顺序并丢弃部分字段来训练,使模型关注字段标签而非位置。
- 在不同字段顺序下,检索性能下降从7.4点降至0.2点,几乎不受影响。
- 适合需要高精度检索的多语言数据发现场景,尤其对低资源语言有效。
我们研究结构化元数据目录中的检索问题,每条记录包含多个字段,用于回答不同类型查询。传统方法先将字段序列化为字符串,强制选择字段顺序。我们发现,这一看似无关紧要的选择在微调后显著影响检索质量:同一索引在不同字段顺序下,标准微调导致nDCG@10下降7.4点,因其依赖绝对位置而非字段标签。为此提出置换不变微调(PI-FT),在数据加载时对每条记录采用随机字段顺序与随机丢弃,使语义绑定于标签而非位置。该改动仅需修改数据加载器约两行代码,对原分布性能影响极小,且将顺序变化带来的损失降至0.2点。实验基于近万项开发统计指标的发现任务,构建了全由大模型生成的跨15语言基准集DevDataBench,覆盖所有指标用于训练与评估。一个118M参数的可本地部署编码器,在nDCG@10上达到0.707,优于所有零样本基线(如text-embedding-3-large的0.556),尤其在低资源语言中提升显著。相关数据、工具链与框架均已开源。
原文摘要 · Abstract (English)
We study retrieval over catalogs of structured metadata, where each record is a small schema whose fields answer different kinds of query. Embedding a record with a text encoder first serializes its fields into a string, which forces a choice of field order. We show this choice, usually treated as an implementation detail, silently controls retrieval quality once the encoder is fine-tuned. A standard fine-tune loses 7.4 nDCG@10 points when the index is rebuilt under a different field order, because it reads absolute position instead of the field labels. We propose permutation-invariant fine-tuning ($\textbf{PI-FT}$), which serializes each record under a freshly sampled field order with random field dropout, so meaning binds to the labels rather than to position. The change is about two lines in the data loader; it costs negligible in-distribution accuracy and cuts the order-change penalty to 0.2 points. We study this in the discovery of development statistics, a catalog of nearly 10,000 indicators that should be searchable in many languages by a model small enough to self-host. As AI assistants and agents increasingly mediate access to public data and statistics, this retrieval step decides whether an answer is grounded in the right indicator or series, making discoverability a precondition for disseminating data through AI. Because usage logs cannot provide training signal for indicators no one has searched, we generate the queries instead. $\textbf{DevDataBench}$ is a fully LLM-generated benchmark of grounded, facet-targeted queries across 15 languages, covering every indicator for both training and evaluation. A fine-tuned 118M-parameter CPU encoder outperforms every zero-shot baseline, including $\texttt{text-embedding-3-large}$ (0.707 vs.\ 0.556 nDCG@10), with the largest gains in low-resource languages. We release the benchmark, pipeline, models, and a reusable PI-FT framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。