RL Studio — Peek LLM

Studio · Computed teaching policy

Fig. 1

Prompt

Start here. Press Train step: the policy samples a group of responses, scores them, and nudges itself toward the better half.

Reward weights

Helpful1.0

Honest1.0

Format0.4

Length0.0

Length is the trap: reward verbosity and the policy learns to pad. Watch true quality part ways with proxy reward.

Algorithm

Group size

G

Step size

η

0.35

KL leash

β

0.08

Train

Sample draws a group without updating — see the variance GRPO's baseline absorbs. Press Space to train one step.

Read the stage

positive advantage — above the group mean negative advantage — below it reference policy

π_{ref}

(frozen SFT)

Computed teaching policy · 8 candidate responses per prompt · exact KL over the catalog

prompt

Policy over all 8 responses

dashed =

π_{ref}

Group relative. The baseline is the group's own average reward — no value model is trained. Rewards and quality are hand-scored features of fixed candidate texts.

Expected under $π$

Proxy reward

E [r]

0.00

0.00true quality

E [q]

proxy rewardtrue quality

0.000

KL (π ‖ π_{ref})

2.08entropy

H (π)

—group mean

\bar{r}

—group std

GRPO in two lines

A_{i} = \frac{r_{i} - \bar{r}}{std (r)}

J = E [\min (ρ A, clip (ρ) A)] - β KL (π_{θ} ‖ π_{ref})

$ρ = π (y) / π_{old} (y)$ is the importance ratio; with clip on, a ratio outside $1 \pm ε$ gets zero gradient and its $ρ$ chip turns rose.

Selected response

Select a card or a policy row to inspect a candidate response.

Why no value model

PPO trains a critic to predict how good each response will be — a second network that can fail on its own. GRPO asks the group instead: sample $G$ answers to the same prompt, and the group's average reward is the baseline. Above average pushes probability up, below average pushes it down. The KL term keeps the tuned policy from drifting off the reference model while it chases reward.