Embedding Studio — Peek LLM

Find a word

Try this first. Search a word, watch the camera dive to it, and see which words live next door.

Analogy: a − b + c

Subtract man from king, add woman — the arithmetic lands on a real word. Directions in this space mean things.

View

Auto-rotate

Drag to orbit · scroll or pinch to zoom · arrow keys rotate · Esc clears.

Selected word

No word selected yet. Each token ID from Concept 01 becomes a row of 768 numbers — a point in a space with 768 axes. This stage is that space, squashed to the 3 axes that matter most.

Read this space

Near = alike. Similarity is the cosine of the angle between two vectors — direction matters, not length.
Directions mean relations. The arrow from man to king is nearly the arrow from woman to queen.
3D is a shadow. PCA keeps the 3 axes of greatest spread; the real geometry lives in 768 dimensions.

\cos (a, b) = \frac{a \cdot b}{‖ a ‖ ‖ b ‖}

Data & sources

The 4,000 most frequent lowercase whole-word tokens from GPT-2's input embedding matrix (wte), verified against an English word list, plus a few demonstration words used in the presets — 4,007 points in total. Reduced with PCA: 3 axes for the view, 64 for the arithmetic. All computation runs offline in this file.

Better Language Models (GPT-2 report)
openai/gpt-2 — model and tokenizer
Mikolov et al. — word2vec
Mikolov et al. — linguistic regularities in word space