用工程化框架让小模型可靠完成学术指导，效果超过大模型聊天机器人。
Harnessing LLMs for Reliable Academic Supervision: A Comparative Study
- 给小模型加符号检索、校验输出等结构化组件，构建可靠决策系统
- 小模型系统在6个维度上评分远超大模型聊天机器人（4.08 vs 1.23）
- 适合对可解释性、可追溯性要求高的学术、医疗等高风险场景
大型语言模型虽能流畅回应单次提示，但将其作为领域决策系统的可靠组件仍面临挑战。解决这一差距需依赖‘支架工程’：在大模型核心周围精心设计确定性结构（符号过滤、检索、类型化输入输出、大模型作为评判者循环、人机协同门控、持久状态、审计日志）。本文以学术指导为案例，对比一个无结构的基准系统ASA（基于GPT-5聊天机器人）与一个多模块系统ASuS（基于更小的GPT-4o-mini，通过LangGraph框架集成符号语义检索、结构化输出验证、有限重试的模型评判、人机协同门控、加权风险评分及每节点的SQLite审计日志）。评估标准聚焦六个支架机制维度（可溯源性、可解释性、一致性、流程完整性、认知负荷、约束遵守）。盲评十位评审员的混合评估结合2×2模型-支架消融实验表明，尽管使用更小的基模型，ASuS在所有维度上均优于ASA。十位评审员平均分分别为4.08和1.23，8位评审员在配对威尔科克森检验中拒绝零假设（α=0.05）。消融实验确认支架结构贡献基本与模型大小无关。研究提炼出七种通用支架工程模式，主张在可靠性、可追溯性、制度一致性高于自由生成流畅性的场景中，支架工程挑战了‘越大越好’的主流认知。
原文摘要 · Abstract (English)
Large language models routinely produce fluent answers to single-shot prompts, yet deploying them as reliable components of a domain decision system is substantially harder. Closing this gap is the work of harness engineering: the deliberate composition of deterministic scaffolding (symbolic filters, retrieval, schema-typed I/O, LLM-as-judge loops, HITL gates, persistent state, audit trails) around an LLM core. We present a case study in academic supervision, a domain combining high-stakes recommendation, longitudinal accountability, and structured operational workflows. We compare a baseline Academic Supervision Assistant (ASA), a GPT-5 chatbot with no scaffolding, against a multi-module system, Academic Supervision System (ASuS) that wraps the much smaller GPT-4o-mini in a LangGraph harness with symbolic semantic retrieval, schema-validated outputs, LLM-as-judge with bounded retry, HITL gates, deterministic weighted risk scoring with LLM narration, and a per-node SQLite audit trail. The evaluation rubric is retargeted at six harness-mechanism dimensions (grounding, explainability, consistency, process integrity, cognitive load, constraint adherence). A blind ten-rater hybrid evaluation, supplemented by a 2 x 2 model-harness ablation, finds that ASuS, despite using a much smaller base model, outscores ASA on every dimension. Across ten raters the pooled mean for ASuS is 4.08 versus 1.23 for ASA, and 8 of 10 raters reject the null at alpha = 0.05 on a paired Wilcoxon test; full numbers are in Sections 6.4 and 6.7. The ablation confirms that the structural contributions of the harness are largely model-invariant. We extract seven recurring harness-engineering patterns and argue that where reliability, traceability, and institutional consistency matter more than open-ended fluency, harness engineering challenges the prevailing 'bigger model is better' intuition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。