用结构化提示提升大模型推理，让其解决物理常识难题
Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem
- 采用STAR框架强制明确任务目标，显著提升推理能力
- 完整架构下准确率达100%，较原始模型提升85个百分点
- 适合需要可靠逻辑推理的智能系统研发者参考
大型语言模型在应对‘汽车清洗问题’这一病毒式传播的推理基准时表现持续不佳，该任务要求模型推断隐含的物理约束。本研究通过变量隔离实验（每组20次，共6组，总计120次试验），考察生产系统中不同提示架构层级对正确推理的影响。使用Claude 3.5 Sonnet并控制超参数（温度0.7，top_p 1.0），发现仅引入STAR（情境-任务-行动-结果）推理框架，即可使准确率从0%提升至85%（p=0.001，Fisher精确检验，优势比13.22）。通过向量数据库检索用户画像上下文可带来额外10个百分点的提升，而RAG上下文再贡献5个百分点，最终在全栈条件下实现100%准确率。结果表明，结构化推理支架——特别是推理前强制目标阐明——对隐含约束推理任务的影响远大于上下文注入。
原文摘要 · Abstract (English)
Large language models consistently fail the "car wash problem," a viral reasoning benchmark requiring implicit physical constraint inference. We present a variable isolation study (n=20 per condition, 6 conditions, 120 total trials) examining which prompt architecture layers in a production system enable correct reasoning. Using Claude 3.5 Sonnet with controlled hyperparameters (temperature 0.7, top_p 1.0), we find that the STAR (Situation-Task-Action-Result) reasoning framework alone raises accuracy from 0% to 85% (p=0.001, Fisher's exact test, odds ratio 13.22). Adding user profile context via vector database retrieval provides a further 10 percentage point gain, while RAG context contributes an additional 5 percentage points, achieving 100% accuracy in the full-stack condition. These results suggest that structured reasoning scaffolds -- specifically, forced goal articulation before inference -- matter substantially more than context injection for implicit constraint reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。