KV Cache Studio — Peek LLM

Peek LLM KV Cache

Studio · Computed decoding trace

Fig. 1

Computed teaching sequence · one representative attention layer

Autoregressive sequence

5 positions

Without cache

recompute history

this step 5 pairs

With KV cache

reuse + append

this step 1 new pair

Work this decode step

5without cache 1with cache K/V projections

4 historical positions reused

Q_{t}

Query is temporary. It reads cached K/V for attention; only the new pair is retained.

Reuse. History stays put while one K/V pair joins the cache. Token K/V projections are counted, not wall-clock speed.

This decode step

5without cache

1with cache

4history reused

4cumulative saved

Counts are token positions whose K/V projections are computed.

Cache shape & memory

Current cache180 KiB

MHAS = 5

FP16 · batch 1 · all 12 layers

Logical K/V shape · one layer[B=1, Hkv=12, S=5, Dh=64]

M_{K V} = 2 L B S H_{K V} D_{h} c

Projected at 2K context72 MiB

The leading 2 accounts for both keys and values. Cache memory grows linearly with sequence length.

Selected token

▁jumps

position 5 · generated token

K/V retained Stored in all 12 layers

Q is used for this decode step, then discarded. K and V remain available to later tokens.

What the cache trades

More memory, less repeated projection. Attention still reads a longer history as the sequence grows; KV cache does not make long-context attention constant-cost.