Can we see inside AI? Interpretability's promise and its limits
Chris Olah and Max Tegmark argue that neural networks can be reverse-engineered into understandable parts, Kokotajlo values readable chains of thought, and Roman Yampolskiy says full explanation of a superhuman system is impossible. Whether models can be inspected determines whether their honesty can ever be verified.
Can we see inside AI models? Show answer
Partly. Researchers can read some internal features and written reasoning, but whether a partial view is enough to verify honesty is disputed.
Why does Chris Olah say models must be reverse-engineered? Show answer
Neural networks are grown through training, not programmed. Nobody wrote down the algorithm, so it has to be recovered from the trained weights.
What did sparse autoencoders make possible? Show answer
They separated the tangled concepts inside neurons into cleaner, readable features, first in a small model and then in a production model.
What finding matters most for checking honesty? Show answer
Features for deception, withholding information and power-seeking were found inside the model. Forcing the deception feature on makes it lie.
Why does Dario Amodei say behavioral testing is not enough? Show answer
Advanced models might sandbag tests or deceive evaluators, so verification needs tools like interpretability kept separate from training.
What does Max Tegmark mean by the halfway point? Show answer
A trained black-box network is only half the job. The second half is turning it into something equally capable that people can read.
What did Tegmark's AI Feynman project do? Show answer
It recovered 100 known physics equations from data alone by training a network and then finding simplifying properties in it.
Why is chain of thought useful, according to Daniel Kokotajlo? Show answer
Language models must output words to carry their thinking forward, so their reasoning, goals and rationalizations can be read.
What threatens chain-of-thought monitoring? Show answer
Models that think without writing everything out, a compressed dialect that drifts into gibberish, and hidden meaning inside innocent-looking text.
Why does Roman Yampolskiy say full explanation is impossible? Show answer
No person can take in an explanation as large as the model. Anything smaller is a lossy compression that can leave out or hide what matters.
What dilemma does Yampolskiy raise about interpretability? Show answer
Progress on explaining a system can also boost its capability, since a system that can read its own workings can improve itself more easily.
What should you not conclude from an AI behaving well in tests? Show answer
That it is honest. A capable system can pass a test while misleading the tester, and may behave differently later.