提出可解释的上下文工程框架,让大模型对齐跨模态实体更高效准确。
Implicit Fine-tuning via Context Engineering: A Curriculum Learning Framework for Multimodal Entity Alignment

- 将微调转化为可解释的上下文设计,通过自适应难度调节实现渐进式对齐。
- 在ICWIKI数据集上,大模型性能差距缩小至0.6%,推理耗时从21小时降至1小时。
- 适合关注大模型高效应用与可解释性研究的研究者参考。
多模态实体对齐(MMEA)旨在识别跨模态等价实体。现有方法依赖黑箱上下文工程提升性能,但受限于大语言模型参数量且缺乏理论可解释性。本文首次从理论上证明上下文工程与模型微调在MMEA任务中的数学等价性,表明提示组件可模拟基于对比学习的序列微调过程。基于此,提出受课程学习启发的PTFEA框架,将微调策略转化为可解释的上下文工程:自适应难度调节通过置信度阈值动态调整信息注入阶段,建立课程学习权重与上下文样本选择的数学对应;三阶段渐进推理从简单到复杂逐步引入实体信息,类比微调中的梯度下降过程。在五个公开数据集上的实验表明,PTFEA持续优于强基线。尤其在ICWIKI数据集上,其使Qwen2.5-72B与14B模型在H@1指标上的差距缩小至0.6%;相较代表性方法MM-ChatAlign,Qwen2.5-72B的推理时间由21小时降至1小时,令牌消耗从2200–3000降至200–400,降幅超80%。本工作首次建立了上下文工程与微调在MMEA中的统一理论框架,为未来将更多微调策略转化为上下文工程范式铺平道路。代码已开源。
原文摘要 · Abstract (English)
Multimodal Entity Alignment (MMEA) aims to identify equivalent entities across different modalities. While existing methods enhance MMEA performance through black-box context engineering strategies, their reliance on LLM parameter capacity and lack of theoretical interpretability remain unresolved. To this end, we first theoretically validate the mathematical equivalence between context engineering and model fine-tuning in MMEA tasks, demonstrating that prompt components simulate contrastive learning-based sequential fine-tuning in MMEA. Building on this foundation, we then propose PTFEA, a curriculum-learning-inspired framework that translates fine-tuning strategies into interpretable context engineering. Specifically, adaptive difficulty modulation dynamically adjusts information injection stages using confidence thresholds, establishing mathematical equivalence between curriculum learning weights and context sample selection; and three-stage progressive inference incorporates entity information from simple to complex cases, mirroring the gradient descent process in fine-tuning. Experiments on five public datasets demonstrate that PTFEA consistently outperforms strong baselines. In particular, on the ICWIKI dataset, PTFEA narrows the H@1 gap between Qwen2.5-72B and 14B to 0.6%. Moreover, compared with the representative context-engineering-based MMEA method MM-ChatAlign, PTFEA reduces the runtime of Qwen2.5-72B from 21 hours to 1 hour and lowers token consumption from 2200-3000 to 200-400, achieving over 80% reduction on the ICWIKI dataset. This work provides the first theoretical framework unifying context engineering and fine-tuning in MMEA, paving the way for future research that seeks to translate additional fine-tuning strategies into context engineering paradigms. Our code is available at https://github.com/DMiC-Lab-HFUT/PTFEA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。