Attention Studio — Peek LLM

Computed from GPT-2 117M

stronger attention masked · cannot look ahead each row looks at the columns · row weights sum to 1

Selected cell

Hover the map — or click to pin a cell. Each cell says how much the row token listens to the column token, in this layer, in this head.

This head's habit

Computed observation · this prompt, this head

Read this map

Score. Every token pair gets a score: query · key, scaled by $\sqrt{d}$ .
Softmax. Each row is squashed into weights that sum to 1.
Mask. The upper triangle is blocked — no looking at future tokens.
Mix. A token's new vector is the weight-blended sum of all values.

Attention (Q, K, V) = softmax (\frac{Q K^{T}}{\sqrt{d_{k}}}) V

Data & sources

Real attention matrices from GPT-2 117M, computed by a verified numpy forward pass on five handcrafted prompts: all 12 layers × 12 heads, pre-softmax scores and post-softmax weights. Runs fully offline.

Attention Is All You Need — Vaswani et al.
openai/gpt-2 — model and tokenizer
The Illustrated Transformer — Jay Alammar