测试大模型在复杂旅游定价中的推理能力，发现其易出错。
PricingLogic: Evaluating LLMs Reasoning on Complex Tourism Pricing Tasks
- 构建300个基于真实定价规则的自然语言问题，分难易两层测试。
- 高难度任务中模型准确率显著下降，暴露出规则理解与计算缺陷。
- 适合关注AI在金融/商务场景可靠性的研究者与开发者。
我们提出PricingLogic，首个评估大语言模型（LLMs）在多重重叠票价规则下可靠自动化旅游定价能力的基准。旅行机构亟需将这一易出错的任务交给AI系统；然而，若未验证可靠性，部署LLMs可能导致重大财务损失并损害客户信任。PricingLogic包含300个基于42个真实定价政策生成的自然语言问题，涵盖两个难度层级：(i) 基础客户类型定价，(ii) 涉及相互作用折扣的打包旅游计算。对一系列LLMs的评估显示，模型在更难任务上性能急剧下降，暴露了规则解读和算术推理中的系统性失败。结果表明，尽管具备通用能力，当前的LLMs在收入关键应用中仍不可靠，需额外防护或领域适配。代码与数据集开源于https://github.com/EIT-NLP/PricingLogic。
原文摘要 · Abstract (English)
We present PricingLogic, the first benchmark that probes whether Large Language Models(LLMs) can reliably automate tourism-related prices when multiple, overlapping fare rules apply. Travel agencies are eager to offload this error-prone task onto AI systems; however, deploying LLMs without verified reliability could result in significant financial losses and erode customer trust. PricingLogic comprises 300 natural-language questions based on booking requests derived from 42 real-world pricing policies, spanning two levels of difficulty: (i) basic customer-type pricing and (ii)bundled-tour calculations involving interacting discounts. Evaluations of a line of LLMs reveal a steep performance drop on the harder tier,exposing systematic failures in rule interpretation and arithmetic reasoning.These results highlight that, despite their general capabilities, today's LLMs remain unreliable in revenue-critical applications without further safeguards or domain adaptation. Our code and dataset are available at https://github.com/EIT-NLP/PricingLogic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。