All Circuits Lead to Rome: Exploring Diversity in Large Model Interpretability

Speaker Introduction

Chen Xi earned a dual‑honors B.Sc. in Computer Science and Mathematics from the University of Toronto, supervised by Professor Gerald Penn. He will begin a Ph.D. in Artificial Intelligence at The Chinese University of Hong Kong (Shenzhen) under Professor Du Mengnan in September 2026. His research focuses on mechanism interpretability and trustworthy AI for large language models.

Research Background

Understanding how large language models (LLMs) perform reasoning, language comprehension, and knowledge retrieval is crucial for transparency, reliability, and safety. Existing mechanism‑interpretability work assumes that each model capability is supported by a single, sparse, irreplaceable internal circuit composed of attention heads, MLP modules, and their connections.

DiscoGP Sheaf Discovery Framework

The talk first revisits the DiscoGP sheaf discovery framework. Unlike conventional circuit discovery that only requires causal relevance, a sheaf must remain functional when all other connections are removed or zero‑ablated. By applying joint graph pruning and flexible‑granularity structural modeling, DiscoGP more precisely locates functional computation paths and addresses combinatorial circuit construction and alternative circuit discovery.

All Circuits Lead to Rome (ICML 2026)

The core of the presentation is the ICML 2026 paper “All Circuits Lead to Rome”. It challenges the hypothesis that a single internal mechanism underlies a given task. The proposed Overlap‑Aware Sheaf Discovery framework (OASR) seeks alternative mechanisms that are as sparse and performant as existing sheafs but structurally dissimilar.

Experiments on Indirect Object Identification (IOI), grammatical judgment, and code‑comment generation show that the same model can yield multiple sheafs with very low structural overlap yet comparable task performance. On the IOI task, two sheafs each achieve 100% accuracy while their edge‑level Jaccard similarity is only 4.1%. As more mechanisms are discovered, the union of their structures expands while their common intersection shrinks.

Implications for Trustworthy AI

The findings indicate that circuit‑discovery methods do not reveal a unique true mechanism but rather a member of a large space of functionally equivalent, partially redundant mechanisms. Recognizing the equivalence, substitutability, stability, and boundary conditions of these mechanisms is essential for model diagnosis, risk identification, behavior intervention, and the broader development of trustworthy AI.