测试环境会骗大模型，导致回答变长但实际能力没提升。
Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B
- 用同一模型对比测试和真实场景的提问方式差异
- 测试提示让模型输出更长，准确率却没明显提高
- 多语言测试发现提示词可能影响模型表现，适合评估者参考
大型语言模型（LLMs）的评测常依赖带有明确评分标准的提示，要求清晰推理和严格格式，而真实部署则需要简洁、契约式回答。我们以单一开源模型（GPT-OSS-20B）为基础，在六组配对实验中固定任务内容与解码策略，仅改变提问框架（评测导向 vs. 真实场景）与推理深度（中/高），涵盖确定性数学、严格代码修复、引用生成、激励反转（谨慎 vs. 能力）、思维链可见性及乌尔都语标题。使用确定性验证器衡量准确率、仅答案合规性、模糊回应/拒答、思维链长度与格式合规性，并预注册差异与综合指标。结果显示，评测框架显著拉长思维链（数百至超1000字符），降低仅答案合规性，但准确率提升有限且不一致。结构化输出中，框架改善了包裹形式（如代码块、列表），但未提升正则验证内容。激励措辞改变错误类型：强调谨慎在高推理下小幅提准并减少自信错误，强调能力则输出更简洁但风险更高。乌尔都语提示同样复现上述现象，且高推理深度时可能降低准确率，揭示多语言评测中的公平性风险。我们提供可复现的A/B框架（提示库、验证器、每轮得分、脚本；带版本DOI），并建议采用中性表述或双框架校验、合同意识评分、风格差异报告、置信度治理及多语言仪表盘，确保评测增益反映真实部署能力。
原文摘要 · Abstract (English)
Benchmarks for large language models (LLMs) often rely on rubric-scented prompts that request visible reasoning and strict formatting, whereas real deployments demand terse, contract-bound answers. We investigate whether such "evaluation scent" inflates measured performance without commensurate capability gains. Using a single open-weights model (GPT-OSS-20B), we run six paired A/B scenarios that hold task content and decoding fixed while varying framing (evaluation-oriented vs. real-world) and reasoning depth (Medium/High): deterministic math, strict code-fix, citation generation, incentive flips (caution vs. competence), CoT visibility, and multilingual (Urdu) headers. Deterministic validators compute accuracy, answer-only compliance, hedging/refusals, chain-of-thought (CoT) length, and schema compliance, with pre-registered deltas and composite indices. Across scenarios, evaluation framing reliably inflates CoT (hundreds to >1000 characters) and reduces answer-only compliance, with limited or inconsistent accuracy gains. In structured outputs, it improves wrappers (e.g., fenced blocks, enumerated lists) but not regex-validated substance. Incentive wording reweights error composition: praising caution modestly improves accuracy at high reasoning and reduces wrong-but-confident errors, whereas praising competence yields terser but riskier outputs. Urdu rubric headers reproduce these signatures and can decrease accuracy at higher reasoning depth, indicating multilingual parity risks. We provide a reproducible A/B framework (prompt banks, validators, per-run scores, scripts; versioned DOI) and practical guidance: neutral phrasing or dual-framing checks, contract-aware grading, style-delta reporting, confidence governance, and multilingual dashboards to ensure that benchmark gains reflect deployable capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。