OpenAI：2026 IH-Challenge ：提升前沿大语言模型指令层次结构的训练数据集研究报告（英文版）（25页）.pdf

1、IH-Challenge:A Training Dataset to Improve Instruction Hierarchyon Frontier LLMsChuan GuoJuan Felipe Ceron UribeSicheng ZhuChristopher A.Choquette-ChooSteph LinNikhil KandpalMilad NasrRai(Michael Pokorny)Sam ToyerMiles WangYaodong YuAlex BeutelKai XiaoOpenAIAbstractInstruction hierarchy(IH)defines h

2、ow LLMs prioritize system,developer,user,and tool instructions under conflict,providing a concrete,trust-ordered policy forresolving instruction conflicts.IH is key to defending against jailbreaks,systemprompt extractions,and agentic prompt injections.However,robust IH behavior isdifficult to train:

3、IH failures can be confounded with instruction-following failures,conflicts can be nuanced,and models can learn shortcuts such as overrefusing.We introduce IH-Challenge,a reinforcement learning training dataset,to addressthese difficulties.Fine-tuningGPT-5-Minion IH-Challenge with online adversarial

4、example generation improves IH robustness by+10.0%on average across 16 in-distribution,out-of-distribution,and human red-teaming benchmarks(84.1%94.1%),reduces unsafe behavior from 6.6%to 0.7%while improving helpfulness ongeneral safety evaluations,and saturates an internal static agentic prompt inj

5、ectionevaluation,with minimal capability regression.We release the IH-Challenge dataset(huggingface.co/datasets/openai/ih-challenge)to support future research on robustinstruction hierarchy.1IntroductionModern LLMs are designed with multiple roles reserved for different input sources,e.g.,systemadmi

6、n,application developer,user,response from tool calls,etc.These roles are also different interms of their trust level.For example,safety guardrails specified by the system admin should notbe overridden by potentially malicious users.Wallace et al.(2024)introduced the notion of theinstruction hierarc