What your attention heatmap isn't telling you

Everyone's staring at attention heatmaps and calling it "interpretability." Almost nobody's asking whether a single attention map actually tells you what the model used to make its decision.

It doesn't. Not on its own. A raw attention map from layer 8 shows you what layer 8 attended to. It says nothing about how that signal got mixed, diluted, or overwritten by every layer before and after it. Attention rollout fixes this — and I built a walkthrough to show why it matters. Here's what makes it more than a "pretty heatmap" demo: Instead of visualizing one layer's attention, I traced how information actually flows through the full transformer stack. → Every layer's attention matrix is extracted, per head, per token → Multi-head attention is averaged, then combined with the residual connection (identity + attention) — this is the step most tutorials skip, and it's the one that actually matters → The combined matrices are matrix-multiplied layer by layer, rolling attention forward from input to output → The result: a single map showing genuine token-to-token influence across the entire network, not just one layer's snapshot

The overlay shows you everything: → Per-layer attention vs. rolled-out attention, side by side → Token importance scores overlaid directly on the input text → A comparison view: which tokens raw attention says "matter" vs. which ones rollout says actually matter → Head-level breakdown so you can see which heads specialize vs. which are noise

No black box. No "trust me, the model looked here." Just linear algebra, applied honestly across every layer instead of cherry-picking one. Built with PyTorch + HuggingFace Transformers + Matplotlib. Runs on any pretrained transformer, fully offline.

⚠️ Important: attention rollout is an approximation, not ground truth. It assumes attention is the primary information pathway, which ignores MLP layers and can still mislead for very deep models. Treat it as a debugging lens, not proof of causality.