01Explainability vs. interpretability

These two words are used almost interchangeably, and the field has never fully agreed on a single definition — Lipton's "The Mythos of Model Interpretability" documents how researchers mean different, sometimes conflicting, things by them. Still, a useful working distinction has settled in. Interpretability is usually about how far a human can understand the model's mechanism directly — easiest when the model is simple and transparent, like a short decision tree or a linear formula. Explainability (XAI) is about producing a human-understandable account of why a model — even an opaque one — produced a given output, often using methods bolted on after training. Government framing reflects the stakes: NIST's Four Principles of Explainable AI and DARPA's XAI program both push for systems that can justify their outputs so people can trust and manage them appropriately.

• Interpretability ≈ you can read the mechanism itself (a transparent model you can follow by hand).
• Explainability ≈ you generate an after-the-fact account of why an opaque model decided what it did.
• The distinction is a convention, not a standard — don't treat any one definition as universally accepted (Lipton, 2016).

02Four questions that classify any method

Almost every explainability technique can be placed by asking four questions. Switch between them to see what each axis means and where common methods land.

03The methods that do the explaining

A handful of post-hoc methods do most of the heavy lifting in practice. LIME (Ribeiro et al., 2016) explains one prediction by sampling perturbed versions of the input, weighting them by closeness, and fitting a simple interpretable surrogate — usually a sparse linear model — whose coefficients become the explanation. SHAP (Lundberg & Lee, 2017) assigns each feature a Shapley value from cooperative game theory: its average marginal contribution across all coalitions of features. SHAP unifies a whole family of additive methods and is the unique solution satisfying local accuracy, missingness, and consistency; TreeSHAP computes it exactly and fast for tree ensembles.

For neural networks, gradient-based methods dominate. Saliency maps (Simonyan et al., 2013) use the gradient of the output with respect to input pixels to highlight influential regions. Integrated Gradients (Sundararajan et al., 2017) integrates gradients along a straight path from a baseline to the actual input, deliberately satisfying two axioms earlier methods broke: Sensitivity (a feature that changes the output gets nonzero attribution) and Implementation Invariance (functionally identical networks get identical attributions). Grad-CAM (Selvaraju et al., 2016/2017) uses gradients flowing into the last convolutional layer to produce a coarse heatmap of where a CNN "looked." Libraries like Captum (PyTorch) and the SHAP package put these in reach of practitioners.

• LIME — perturb around one input, fit a local surrogate; its coefficients are the explanation (model-agnostic, local).
• SHAP — game-theoretic Shapley values; additive, consistent feature attributions; TreeSHAP is exact for trees.
• Saliency / Integrated Gradients / Grad-CAM — gradient-based, model-specific attributions for neural nets and CNNs.
• Permutation importance — shuffle a feature, measure the score drop; a model-agnostic global view (scikit-learn).

04See it: which features drove this decision?

Below is a simulated loan-approval model that has denied one applicant. There is no real model here — the contributions are illustrative, hand-set to show how an attribution explanation reads. Each bar shows how much a feature pushed the decision toward approve (right) or deny (left). Switch the method to watch the same prediction get re-weighted: different techniques can emphasise different features and even disagree — which is exactly why a single explanation is evidence, not proof.

05Opening the box: mechanistic interpretability

Attribution tells you which inputs mattered. Mechanistic interpretability — a program associated with Chris Olah and the Distill Circuits thread — goes further: it tries to reverse-engineer the network's internal computation into human-understandable parts. In this framing, features are directions in activation space that correspond to meaningful concepts, and circuits are the weighted connections between features that implement a computation.

The hard obstacle is superposition (Anthropic, Toy Models of Superposition, 2022): a network represents more features than it has dimensions by packing them into overlapping, non-orthogonal directions. That produces polysemantic neurons — single neurons that fire for several unrelated concepts — so you can't just read one neuron and know what it means. Sparse autoencoders / dictionary learning (Towards Monosemanticity, 2023; Scaling Monosemanticity, 2024) partly undo this by decomposing activations into a much larger set of sparsely active, mostly monosemantic features — and the 2024 work applied it to a production model, Claude 3 Sonnet, extracting millions of interpretable features. More recently, attribution graphs / circuit tracing (Anthropic, 2025) use cross-layer transcoders to map how information flows from input tokens through intermediate features to outputs, revealing multi-step internal reasoning. These are first-party research results and an active, fast-moving area — partial and model-specific, not a finished account of any model.

• Features & circuits — meaningful directions in activation space, wired together to implement computations.
• Superposition makes neurons polysemantic, which is why per-neuron reading fails; sparse autoencoders recover cleaner features.
• Attribution graphs trace internal multi-step reasoning — but findings are partial, model-specific, and self-reported.

06The one trap: a good explanation can still be wrong

This is the most important thing to carry away. NIST's four principles include Explanation Accuracy precisely because an explanation can be plausible yet unfaithful — it can sound convincing while not reflecting what the model actually did. Different attribution methods can disagree on the very same prediction. Saliency maps can be visually compelling and still fail to reveal the spurious correlation a model is really exploiting (Google PAIR's saliency explorable shows exactly this). And attention weights are not a reliable explanation: Jain & Wallace (2019) found attention often diverges from gradient-based importance, and that very different attention patterns can yield identical predictions — a point that is itself contested by later rebuttals, so treat attention-as-explanation as an open debate. The practical rule: treat any single explanation as one piece of evidence, not a verdict, and remember a feature mattering to the model is not the same as it mattering in the world.

• Faithfulness is not guaranteed — post-hoc explanations approximate the model (NIST "Explanation Accuracy" exists for this reason).
• Methods disagree — cross-check, and don't trust a lone saliency map to expose a hidden bias.
• Attention ≠ explanation (Jain & Wallace, 2019) — and even that claim is debated; present it as contested.

07Check your understanding

08Take it with you & go deeper