检验文本生成音乐是否真听指令，发现多数模型其实没听懂。
Do Text-to-Music Models Really Follow Instructions? A Counterfactual Evaluation of Key and Beat Grouping
- 用反事实对比法分离指令影响与固有偏好
- 稳定音频3仅在调性上真听指令，节拍控制多是惯性输出
- 适合评估生成模型真实可控性的研究者使用
提示属性一致常被当作文本到音乐可控性的证据，但目标属性出现可能只是因其在模型输出分布中本就常见。本文提出匹配的反事实评估方法，将目标出现与指令可归因控制分开。每组包含一个省略目标属性的中性输入，以及两个其他条件相同但交换请求目标的输入。三者均通过冻结的原生接口适配器以共享种子生成。应用于三个开源系统中的全局调性和节拍分组，该设计改变了原有结论：ACE-Step 1.5和稳定音频3中等版本表现出显著调性控制，而LeVo2则无；对于节拍分组，相同模型会转向罕见的三拍目标，但高四拍一致性主要源自中性输出——稳定音频3在0.97的中性情况下已呈现四拍结构，而在明确要求四拍时仅达0.56。非目标安慰剂、外部识别器验证、盲评专家标注及多种子哨兵均支持归因结论。当目标输出先验不同时，一致性描述模型实际生成内容，而匹配的中性与目标互换对比才能检验指令是否真正改变结果。
原文摘要 · Abstract (English)
Prompted attribute agreement is widely used as evidence of text-to-music controllability, yet a requested attribute may occur simply because it is already common in the model's output distribution. We introduce a matched counterfactual evaluation that separates target occurrence from instruction-attributable control. Each family contains a neutral input that omits the scored attribute and two otherwise matched inputs that swap the requested target. All three are rendered through frozen native-interface adapters with a shared seed. Applied to global key and beat grouping in three open systems, this design changes the empirical conclusion. ACE-Step 1.5 and Stable Audio 3 Medium exhibit substantial key control, whereas LeVo2 does not. For beat grouping, the same models redirect toward the rare three-beat target, but high four-beat agreement is largely inherited from neutral outputs: Stable Audio 3 produces four-beat grouping in 0.97 of neutral cases but only 0.56 under its explicit four-beat treatment. Off-attribute placebos, external recognizer validation, blind expert annotation, and multi-seed sentinels support the attribution. When targets have unequal output priors, agreement describes what a model produced, while matched neutral and target-swap contrasts test whether the instruction changed it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。