优化法律文本结构与提示设计，可提升大模型问答准确率20个百分点。
The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting
- 用Markdown等显式格式化法律文本，增强模型理解。
- 结构化输入使GPT-4.1准确率提升约20个百分点。
- 精心设计系统提示可再增10-13个百分点，适合法律智能应用。
法律合同具有内在的、语义关键的结构（如章节、条款），对人类理解至关重要，但其对大语言模型处理的影响尚未充分探索。本文研究了显式输入结构和提示工程对GPT-4o与GPT-4.1在CUAD数据集片段上的法律问答任务表现的影响。对比了多种输入格式：原始结构化纯文本（人工生成自CUAD）、去除换行的纯文本、Azure OCR提取的纯文本、GPT-4o Vision提取的纯文本，以及由GPT-4o Vision解析并解释生成的Markdown格式。通过将任务指令移至系统提示，并明确告知模型输入的结构特性，评估提示工程效果。结果表明，GPT-4o对输入结构变化具有较强鲁棒性，但整体性能有限；而GPT-4.1对结构敏感，差结构输入表现不佳（与GPT-4o相当），良好结构输入（原CUAD文本、GPT-4o Vision文本及Markdown）可使其准确率提升约20个百分点。进一步优化系统提示，加入任务细节和结构提醒，可使准确率再提高约10-13个百分点，最终在最佳条件下达到79%的精确匹配准确率。本研究实证表明，尽管新模型更具韧性，但精心设计的输入结构与提示策略仍是优化大模型在高风险法律场景下表现的关键。
原文摘要 · Abstract (English)
Legal contracts possess an inherent, semantically vital structure (e.g., sections, clauses) that is crucial for human comprehension but whose impact on LLM processing remains under-explored. This paper investigates the effects of explicit input text structure and prompt engineering on the performance of GPT-4o and GPT-4.1 on a legal question-answering task using an excerpt of the CUAD. We compare model exact-match accuracy across various input formats: well-structured plain-text (human-generated from CUAD), plain-text cleaned of line breaks, extracted plain-text from Azure OCR, plain-text extracted by GPT-4o Vision, and extracted (and interpreted) Markdown (MD) from GPT-4o Vision. To give an indication of the impact of possible prompt engineering, we assess the impact of shifting task instructions to the system prompt and explicitly informing the model about the structured nature of the input. Our findings reveal that GPT-4o demonstrates considerable robustness to variations in input structure, but lacks in overall performance. Conversely, GPT-4.1's performance is markedly sensitive; poorly structured inputs yield suboptimal results (but identical with GPT-4o), while well-structured formats (original CUAD text, GPT-4o Vision text and GPT-4o MD) improve exact-match accuracy by ~20 percentage points. Optimizing the system prompt to include task details and an advisory about structured input further elevates GPT-4.1's accuracy by an additional ~10-13 percentage points, with Markdown ultimately achieving the highest performance under these conditions (79 percentage points overall exact-match accuracy). This research empirically demonstrates that while newer models exhibit greater resilience, careful input structuring and strategic prompt design remain critical for optimizing the performance of LLMs, and can significantly affect outcomes in high-stakes legal applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。