用符号规则+可解释性，发现模型靠记忆而非理解。
Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation
- 结合符号规则与机制可解释性，设计算法级评测
- 记忆型模型在未见数据上仍达94%准确率
- 能揭示模型是否真正掌握语义规则，适合研究者
基于准确率的评估无法可靠区分真正的泛化与记忆、泄露或脆弱启发式等捷径，尤其在小数据场景下。本文主张采用机制感知的评估方法，将任务相关的符号规则与机制可解释性相结合，生成算法级通过/失败评分，精准揭示模型在何处实现泛化、何处利用模式。我们在NL-to-SQL任务上验证：两个相同架构的模型，一个无模式信息（强制记忆），一个有模式信息（支持语义锚定）。标准评估显示记忆模型在未见数据上达到94%字段名准确率，看似表现良好；但符号-机制评估发现其违反核心模式泛化规则，这一缺陷在准确率指标中完全不可见。
原文摘要 · Abstract (English)
Accuracy-based evaluation cannot reliably distinguish genuine generalization from shortcuts like memorization, leakage, or brittle heuristics, especially in small-data regimes. In this position paper, we argue for mechanism-aware evaluation that combines task-relevant symbolic rules with mechanistic interpretability, yielding algorithmic pass/fail scores that show exactly where models generalize versus exploit patterns. We demonstrate this on NL-to-SQL by training two identical architectures under different conditions: one without schema information (forcing memorization), one with schema (enabling grounding). Standard evaluation shows the memorization model achieves 94% field-name accuracy on unseen data, falsely suggesting competence. Our symbolic-mechanistic evaluation reveals this model violates core schema generalization rules, a failure invisible to accuracy metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。