The Misery of Mechanistic Interpretability: A Formal Perspective

Tobias Ladner, Matthias Althoff

2026 · arXiv (Cornell University)

Mechanistic interpretability has become the dominant lens for understanding frontier language models, as their inner workings are complex and inherently black boxes. To gain insights into these models, interpretable replacement networks (IRNs) are trained at all layers, exposing interpretable features through sparsely activated neurons. However, the faithfulness of an IRN is usually evaluated only empirically on clean data, and we show that even semantically minor input perturbations flip the dominant IRN features-and thus the human-understandable interpretation-across five open-weight model…

Read the paper →

Explore this paper's citation graph on Constellation.