Neural Networks Are Surprisingly Interpretable: New Research Challenges Black Box Assumptions

Former OpenAI interpretability researcher Nick Cammarata published a paper titled "Neural Networks Are Already Clean" (https://already-clean.goodfire.com/already-clean.pdf), which OpenAI CEO Sam Altman publicly endorsed. The work re-examines the most pessimistic evidence in mechanistic interpretability and concludes that the apparent chaos in neural networks largely stems from early architectural choices and limited observation tools, not from inherent conceptual entanglement.

Qwen3-32B Audit: ~90% of Neurons Have a Single Primary Function

An initial audit of Qwen3-32B found that approximately 90% of neurons exhibit one clear primary function at their positive and negative activation extremes, usually with only related side branches. About 7% have two related functions, and only 2–4% remain unclear. For example, neurons activated by concepts like "Titanic," "Baroque," or "Christmas" cluster tightly around their core semantics rather than firing randomly.

The author cautions that the 90% figure reflects the audit's specific criteria and does not mean the entire model is fully explained.

Vision Models: Modern Architectures Achieve 99–100% Neuron Interpretability

InceptionV1's deep-layer neurons were long considered a hallmark of confusion. With older tools, only about 5% of final-layer neurons were interpretable; better tools raised this to roughly 50% for the same network. Crucially, when modern components — residual connections and LayerNorm — are added while keeping parameter count and training data comparable (ConvNeXt), interpretability jumps to 99–100% of final-layer neurons. This order-of-magnitude change occurs without scaling the model or changing the data, making it difficult to attribute the previous opacity to inherent neuron polysemy.

Reinterpreting the Famous "Cat-Car" Neuron

InceptionV1's neuron 4e:55, which responds to both cat faces and car fronts, has been a classic example of unrelated concepts forced into one unit. Re-examining its inputs, downstream connections, and positive/negative responses reveals a shared geometric motif: sharp V-shapes or bilateral pointed structures. Cat and fox snouts, car fronts, pool tables, and the Shinkansen nose all provide this pattern, while round shapes drive negative activation. The neuron still participates in both cat and car representations, but the combination is not arbitrary — the earlier polysemy likely reflected the observer's failure to find the shared pattern.

Neural Networks Are Already Clean
https://already-clean.goodfire.com/already-clean.pdf