arXiv:2605.00400cs.IRcs.CL2026-05中稿 · ed被引 1

提出新任务与数据集,让模型更懂用户细粒度指令。

FollowTable: A Benchmark for Instruction-Following Table Retrieval

论文配图:FollowTable: A Benchmark for Instruction-Following Table Retrieval
图 1 · 摘自论文原文
  • 定义指令遵循型表格检索新任务,需兼顾主题相关与指令约束。
  • 在FollowTable上测试发现现有模型严重依赖表面语义,忽略结构约束。
  • 设计响应度评分指标,量化模型对指令变化的适应能力,适合评估智能体系统。

表格检索传统上被当作一次性检索问题,相关性主要由主题语义相似性决定。随着大模型驱动的智能体系统普及,对结构化数据的访问日益依赖于具体指令,相关性不仅取决于主题匹配,还需满足显式的内容与模式约束。为此,我们正式提出指令遵循型表格检索(IFTR)这一新任务,要求模型同时满足主题相关性和细粒度指令约束。我们识别出两大核心挑战:(i) 对内容范围敏感,如包含/排除约束;(ii) 对基于模式的要求具备认知能力,包括列语义和表示粒度——这些能力在现有检索器中普遍缺失。为支持系统评估,我们构建了首个大规模基准 FollowTable,采用分类驱动的标注流程。我们还提出新指标——指令响应度得分,用于评估检索排序是否相对于仅基于主题的基线,持续响应用户指令的变化。结果表明,现有检索模型在处理表格数据的细粒度指令时表现不佳,普遍存在对表面语义线索的系统性偏倚,且在应对模式约束方面仍受限,凸显未来改进空间。

原文摘要 · Abstract (English)

Table Retrieval (TR) has traditionally been formulated as an ad-hoc retrieval problem, where relevance is primarily determined by topical semantic similarity. With the growing adoption of LLM-based agentic systems, access to structured data is increasingly instruction-driven, where relevance is conditional on explicit content and schema constraints rather than topical similarity alone. We therefore formalize Instruction-Following Table Retrieval (IFTR), a new task that requires models to jointly satisfy topical relevance and fine-grained instruction constraints. We identify two core challenges in IFTR: (i) sensitivity to content scope, such as inclusion and exclusion constraints, and (ii) awareness of schema-grounded requirements, including column semantics and representation granularity--capabilities largely absent in existing retrievers. To support systematic evaluation, we introduce FollowTable, the first large-scale benchmark for IFTR, constructed via a taxonomy-driven annotation pipeline. We further propose a new metric, termed the Instruction Responsiveness Score, to evaluate whether retrieval rankings consistently adapt to user instructions relative to a topic-only baseline. Our results indicate that existing retrieval models struggle to follow fine-grained instructions over tabular data. In particular, they exhibit systematic biases toward surface-level semantic cues and remain limited in handling schema-grounded constraints, highlighting substantial room for future improvements.

表格检索指令遵循智能体评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。