解决大模型结构化提取的格式不一致问题，提升生产级可靠性。
PromptPort: A Reliability Layer for Cross-Model Structured Extraction
- 通过确定性归一化+轻量验证器，统一不同模型输出格式。
- 在6个模型家族上提升F1 6-16点，接近最优性能。
- 无需修改基础模型，支持未知模型部署，适合生产系统使用。
利用大模型进行结构化提取在实际应用中失败，并非因理解能力不足，而是输出格式在不同模型和提示间极不可靠。一个在GPT-4上返回干净JSON的提示，在Llama上可能输出带代码块、散文包裹或格式错误的内容，导致严格解析器拒绝本应正确的结果。我们将其定义为格式坍塌，并提出双指标评估框架：ROS（严格解析，衡量运行可靠性）与CSS（归一化后，衡量语义能力）。在包含37,346个样本的相机元数据基准上，六种模型家族均出现严重格式坍塌（例如，Gemma-2B：ROS 0.116 vs CSS 0.246），跨模型可移植性差距显著（F1 0.4~0.6）。随后提出PromptPort，一种结合确定性归一化、轻量级验证器（DistilBERT）与安全覆盖策略的可靠性层。该方法恢复格式错误（+6至8 F1），通过验证器驱动语义选择（比归一化再+14至16 F1），并逼近逐字段原地最优表现（0.890对比0.896零样本），且无需修改基础模型。该方法可泛化至未见模型族，不确定时明确拒答，支持生产环境可靠部署。
原文摘要 · Abstract (English)
Structured extraction with LLMs fails in production not because models lack understanding, but because output formatting is unreliable across models and prompts. A prompt that returns clean JSON on GPT-4 may produce fenced, prose-wrapped, or malformed output on Llama, causing strict parsers to reject otherwise correct extractions. We formalize this as format collapse and introduce a dual-metric evaluation framework: ROS (strict parsing, measuring operational reliability) and CSS (post-canonicalization, measuring semantic capability). On a 37,346-example camera metadata benchmark across six model families, we find severe format collapse (for example, Gemma-2B: ROS 0.116 versus CSS 0.246) and large cross-model portability gaps (0.4 to 0.6 F1). We then present PromptPort, a reliability layer combining deterministic canonicalization with a lightweight verifier (DistilBERT) and a safe-override policy. PromptPort recovers format failures (plus 6 to 8 F1), adds verifier-driven semantic selection (plus 14 to 16 F1 beyond canonicalization), and approaches per-field oracle performance (0.890 versus 0.896 in zero-shot) without modifying base models. The method generalizes to held-out model families and provides explicit abstention when uncertain, enabling reliable structured extraction in production deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。