arXiv:2607.22115cs.DBcs.AI2026-07中稿 · ACM SIGMOD 2027

为文本转SQL测试引入真实角色访问控制,揭示大模型在权限受限下的实际表现短板。

Benchmarking Text-to-SQL under Role-Based Access Control

论文配图:Benchmarking Text-to-SQL under Role-Based Access Control
图 1 · 摘自论文原文
  • 用大模型自动生成符合数据库结构的角色权限,模拟真实场景的访问限制。
  • 在有权限约束时,多数高分模型因频繁违规导致性能骤降。
  • 提供可识别权限冲突的评估指标,适合关注实际部署的开发者使用。

给定数据库S和自然语言问题Q,文本转SQL系统旨在生成正确回答Q的SQL查询。当前主流基准大多假设无限制访问S,但现实中用户访问常受角色基于访问控制(RBAC)政策限制。这导致基准结果与真实性能脱节:高分大模型可能在权限环境中频繁违反RBAC,或拒绝仅使用授权数据即可回答的问题。为此,我们提出一个包含真实RBAC约束的综合性文本转SQL评估框架,采用大模型辅助流程,在现有基准上自动添加合理用户角色与访问策略。我们把角色生成建模为对数据库模式的结构化推理过程,大模型先从模式推断应用上下文,再生成与上下文一致的角色职责与访问范围,并由领域专家进行人工质量审核。该框架还包含能识别RBAC特异性失败模式的评估指标,区分查询实用性与权限合规性。我们在多个主流基准上应用此框架,对前沿文本转SQL系统进行系统性实证研究。结果显示,许多在无约束条件下得分高的模型(尤其是开源大模型)在加入权限限制后性能显著下降,主要由于频繁的RBAC违规。

原文摘要 · Abstract (English)

Given a database S and a natural language question Q, text-to-SQL systems aim to generate an SQL query that correctly answers Q when executed against S. Currently, popular text-to-SQL benchmarks mostly assume unrestricted access to S; in practice, however, user access is often restricted, e.g., through role-based access control (RBAC) policies. This leads to a potential disconnect between benchmarking results and real-world performance: an LLM with high benchmark scores might perform poorly in an access-controlled environment, by frequently violating RBAC, or rejecting a query q that could be answered with only permitted data in S. Motivated by this, we present a comprehensive text-to-SQL benchmarking framework with realistic RBAC constraints, which features an LLM-assisted workflow that augments existing text-to-SQL benchmarks with plausible user roles and access policies. To do so, we formulate the problem of role synthesis as a structured reasoning process over the database schema, in which the LLM first infers the application context from the schema, and then derives role responsibilities and access scopes consistent with this context. This process is audited by human-in-the-loop quality control, in which domain experts perform metric-guided screening on the generated roles. Besides the augmented dataset, the proposed framework also contains evaluation metrics that identify RBAC-specific failure modes, and disentangle SQL utility from access-control compliance. We apply the proposed framework to several widely-used benchmarks, and conduct a systematic empirical study of state-of-the-art text-to-SQL systems. The results show that many solutions (especially open-weight LLMs) with high benchmarking scores under an unrestricted setting suffer sharp performance degradation once access constraints are in place, due to frequent RBAC violations.

文本转SQL权限控制大模型评估基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。