Can we see inside AI? Interpretability's promise and its limits
October 6, 2026 · 20 min read
Chris Olah and Max Tegmark argue that neural networks can be reverse-engineered into understandable parts, Kokotajlo values readable chains of thought, and Roman Yampolskiy says full explanation of a superhuman system is impossible. Whether models can be inspected determines whether their honesty can ever be verified.
The rules, numbers and steps worth keeping in front of you.
- We can see inside AI partly; the dispute is whether partial is enough.
- Olah: neural networks are grown, not programmed, so their workings must be reverse-engineered.
- Olah: neurons mix concepts; sparse autoencoders separate them into readable features.
- Olah: a deception feature exists in a production model, and forcing it on makes the model lie.
- Olah: much is still unobserved, so no visible deception signal does not prove honesty.
- Amodei: models that may deceive evaluators need inspection tools kept separate from training.
- Tegmark: past harm from automation came from trusting systems nobody understood.
- Tegmark: distill what a network learned into compact code that can be read and verified; proof tools do not yet work on neural networks.
- Kokotajlo: models reason in written words, which makes their thinking readable for now.
- Kokotajlo: that window closes if models think without writing or hide meaning in text.
- Yampolskiy: any usable explanation is a lossy summary; the only true one is the model itself.
- Yampolskiy: proofs fail for self-modifying software, and 99.9 percent is never 100.
- Yampolskiy: use narrow systems, which can be understood pretty well.
- No source reports a tool that has certified a frontier model as honest.
This is the short form. The full insight has every expert's reasoning, their own words and the sources.