OpenAI：2025权重稀疏Transformer具有可解释电路研究报告（英文版）（31页）.pdf

1、Weight-sparse transformers have interpretable circuitsLeo Gao1Achyuta Rajaram1Jacob Coxon1Soham V.Govande1Bowen Baker1Dan Mossing1AbstractFinding human-understandable circuits in lan-guage models is a central goal of the fi eld ofmechanistic interpretability.We train models tohave more understandabl

2、e circuits by constrain-ing most of their weights to be zeros,so that eachneuron only has a few connections.To recoverfi ne-grained circuits underlying each of severalhand-crafted tasks,we prune the models to isolatethe part responsible for the task.These circuitsoften contain neurons and residual c

3、hannels thatcorrespond to natural concepts,with a small num-ber of straightforwardly interpretable connectionsbetween them.We study how these models scaleand fi nd that making weights sparser trades off ca-pabilityforinterpretability,andscalingmodelsizeimproves the capability-interpretability fronti

4、er.However,scaling sparse models beyond tens ofmillions of nonzero parameters while preservinginterpretability remains a challenge.In addition totraining weight-sparse models de novo,we showpreliminary results suggesting our method canalso be adapted to explain existing dense models.Our work produce

5、s circuits that achieve an un-precedented level of human understandability andvalidates them with considerable rigor.1.IntroductionWhile neural networks,such as large language models,haverapidly increased in capability in recent years,we still un-derstand very little about how they work.Mechanistic

6、in-terpretability seeks to reverse engineer neural networks andfully understand the algorithms they implement internally.A major diffi culty for interpreting transformers is that theactivations and weights are not directly comprehensible;for example,neurons activate in unpredictable patterns thatdon