Original paper link
LLMs scaled to a sufficient size can exhibit reasoning abilities. However, achieving these capabilites in training demands a lot of computation. Another results we have is that LLMs can be effectively augmented using Chain of thought (COT) , this involves carefully designed few shot prompts or prompts such as “think step by step” , enables models to product intermediate reasoning steps, there by solving complex tasks. But … Despited their effectiveness, these techniques have their limitations which is dependence on human. Furthermore, by constraining LLMs to replicate human thought process, we are constraining them by human-provided exemplars, thereby preventing them from exploration of non-human superior reasoning pathways.
To tackle this, Deepseek uses GRPO
In simple reinforcement learning, our setup looks something like:
┌──────────────────┐
┌───>│ Agent (LLM) │──────┐ State (S_t) {input so far}
│ └──────────────────┘ │ Action (A_t) {next token}
Reward (r_t) │
│ ┌──────────────────┐ │
└────│ Env (tokens) │<─────┘
└──────────────────┘
Policy: π θ ( a t ∣ s t ) \pi_\theta(a_t \mid s_t) π θ ( a t ∣ s t ) → {probability of next token}
So, in this setup, our LLM learns from the reward function and performs accordingly so that reward is maximized.
max θ J ( θ ) = E τ ∼ π θ [ R ( τ ) ] \max_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[R(\tau)\right] max θ J ( θ ) = E τ ∼ π θ [ R ( τ ) ]
J ( θ ) J(\theta) J ( θ ) → objective function
τ \tau τ → trajectory
π θ \pi_\theta π θ → policy
R ( τ ) R(\tau) R ( τ ) → cumulative rewards
**Policy is used to produce a probability distructions over all actions/tokens
J ( θ ) = E [ R ( τ ) ] J(\theta) = \mathbb{E}[R(\tau)] J ( θ ) = E [ R ( τ )]
∇ θ J ( θ ) = ∇ θ E [ R ( τ ) ] \nabla_\theta J(\theta) = \nabla_\theta \, \mathbb{E}[R(\tau)] ∇ θ J ( θ ) = ∇ θ E [ R ( τ )]
∇ θ J ( θ ) = ∇ θ ∫ π θ ⋅ R ( τ ) d τ \nabla_\theta J(\theta) = \nabla_\theta \int \pi_\theta \cdot R(\tau) \, d\tau ∇ θ J ( θ ) = ∇ θ ∫ π θ ⋅ R ( τ ) d τ
⇒ ∇ θ J ( θ ) = ∫ ∇ θ π θ R ( τ ) d τ \Rightarrow \nabla_\theta J(\theta) = \int \nabla_\theta \pi_\theta \, R(\tau) \, d\tau ⇒ ∇ θ J ( θ ) = ∫ ∇ θ π θ R ( τ ) d τ
Using the log-derivative trick:
∇ θ log π θ = ∇ θ π θ π θ ⇒ ∇ θ π θ = π θ ∇ θ log π θ \nabla_\theta \log \pi_\theta = \frac{\nabla_\theta \pi_\theta}{\pi_\theta} \;\Rightarrow\; \nabla_\theta \pi_\theta = \pi_\theta \, \nabla_\theta \log \pi_\theta ∇ θ log π θ = π θ ∇ θ π θ ⇒ ∇ θ π θ = π θ ∇ θ log π θ
⇒ ∇ θ J ( θ ) = ∫ π θ ∇ θ log π θ R ( τ ) d τ \Rightarrow \nabla_\theta J(\theta) = \int \pi_\theta \, \nabla_\theta \log \pi_\theta \, R(\tau) \, d\tau ⇒ ∇ θ J ( θ ) = ∫ π θ ∇ θ log π θ R ( τ ) d τ
⇒ ∇ θ J ( θ ) = E [ ( ∇ θ log π θ ) R ( τ ) ] \Rightarrow \nabla_\theta J(\theta) = \mathbb{E}\left[(\nabla_\theta \log \pi_\theta) \, R(\tau)\right] ⇒ ∇ θ J ( θ ) = E [ ( ∇ θ log π θ ) R ( τ ) ]
Here ∇ θ J ( θ ) \nabla_\theta J(\theta) ∇ θ J ( θ ) is the gradient .
Update the parameter as:
θ ← θ + α ⋅ ∇ θ J ( θ ) \theta \leftarrow \theta + \alpha \cdot \nabla_\theta J(\theta) θ ← θ + α ⋅ ∇ θ J ( θ )
This is the standard REINFORCE algorithm .
In general case it is:
E τ ∼ π θ [ ⋅ ] \mathbb{E}_{\tau \sim \pi_\theta}[\cdot] E τ ∼ π θ [ ⋅ ]
(where π θ \pi_\theta π θ is the policy and τ \tau τ is the trajectory)
In case of LLMs, it converts as:
E x ∼ D [ E y ∼ π θ ( ⋅ ∣ x ) [ ⋅ ] ] \mathbb{E}_{x \sim D}\left[\mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[\cdot]\right] E x ∼ D [ E y ∼ π θ ( ⋅ ∣ x ) [ ⋅ ] ]
x x x → prompt
D D D → set of prompts
y y y → response
π θ ( ⋅ ∣ x ) \pi_\theta(\cdot \mid x) π θ ( ⋅ ∣ x ) → response distribution given by LLM
Given a prompt, sample a response y ∼ π θ ( ⋅ ∣ x ) y \sim \pi_\theta(\cdot \mid x) y ∼ π θ ( ⋅ ∣ x ) ← Expensive step
Evaluate completed response: R = R ( x , y ) R = R(x, y) R = R ( x , y )
Compute log probability gradient: g = R ∇ θ log π θ ( y ∣ x ) g = R \, \nabla_\theta \log \pi_\theta(y \mid x) g = R ∇ θ log π θ ( y ∣ x )
Update parameter as: θ ← θ + α ⋅ g \theta \leftarrow \theta + \alpha \cdot g θ ← θ + α ⋅ g
Gradients are noisy.
No explicit token-level credit. A correct answer doesn’t mean that all tokens were equally good.
No constraints on policy updation. The final answer can deviate a lot!
Original paper link
GRPO is the RL framework for Deepseek. It was originally proposed to simplify the training process and reduce the resource consumption of PPO
The proposed solution for the challenges mentioned above is PPO .
PPO : J ( θ ) = Maximize advantages ⏟ + Don’t deviate too much from before ⏟ Proximal part \text{PPO}: \; J(\theta) = \underbrace{\text{Maximize advantages}}_{} \;+\; \underbrace{\text{Don't deviate too much from before}}_{\text{Proximal part}} PPO : J ( θ ) = Maximize advantages + Proximal part Don’t deviate too much from before
For a probability distribution given by the policy, we don’t have a target distribution to compare it. PPO finds a so-called “scaling factor” for these distributions denoted by A t A_t A t .
a t ∼ π θ ( s t ) { sample action from probability } a_t \sim \pi_\theta(s_t) \quad \{\text{sample action from probability}\} a t ∼ π θ ( s t ) { sample action from probability }
P ( a t ) = π θ ( a t ∣ s t ) P(a_t) = \pi_\theta(a_t \mid s_t) P ( a t ) = π θ ( a t ∣ s t )
We want to scale up this probability by A t A_t A t .
Update: ∇ θ π θ ( a t ∣ s t ) ⋅ A t \nabla_\theta \, \pi_\theta(a_t \mid s_t) \cdot A_t ∇ θ π θ ( a t ∣ s t ) ⋅ A t
Loss = π θ ( a t ∣ s t ) ⋅ A t \text{Loss} = \pi_\theta(a_t \mid s_t) \cdot A_t Loss = π θ ( a t ∣ s t ) ⋅ A t
Total loss = ∑ t = 0 T π θ ( a t ∣ s t ) ⋅ A t \text{Total loss} = \sum_{t=0}^{T} \pi_\theta(a_t \mid s_t) \cdot A_t Total loss = ∑ t = 0 T π θ ( a t ∣ s t ) ⋅ A t
Mathematically convenient to add a log:
Total loss = ∑ t = 0 T log π θ ( a t ∣ s t ) ⋅ A t \text{Total loss} = \sum_{t=0}^{T} \log \pi_\theta(a_t \mid s_t) \cdot A_t Total loss = ∑ t = 0 T log π θ ( a t ∣ s t ) ⋅ A t
PPO defines A t A_t A t as “How much better than the average was the action we took?”
→ To answer this, we introduce a Value function V ϕ V_\phi V ϕ .
V ϕ V_\phi V ϕ is another model used by the agent, separate from the LLM, with its own weights ϕ \phi ϕ .
Agent: (S_0) --A_0,a_0--> (S_1) --A_1,a_1--> ... --A_{T-1},a_{T-1}--> (S_{T-1}) --A_T,a_T--> (S_T) Env
V_φ(S_0) V_φ(S_1) V_φ(S_{T-1}) V_φ(S_T)
V ϕ V_\phi V ϕ → tells how good the response so far is regardless of what we predict next.
V ( S t ) = E τ ∼ π [ r t + r t + 1 + ⋯ + r T ∣ S t ] { Expectation of future rewards } V(S_t) = \mathbb{E}_{\tau \sim \pi}\left[r_t + r_{t+1} + \dots + r_T \mid S_t\right] \quad \{\text{Expectation of future rewards}\} V ( S t ) = E τ ∼ π [ r t + r t + 1 + ⋯ + r T ∣ S t ] { Expectation of future rewards }
Computing this is impossible as it would result in a ridiculously large tree. So, instead of calculating, our model uses agent V ϕ V_\phi V ϕ to estimate it.
We can also add a discount factor:
V ( S t ) = E τ ∼ π [ r t + γ r t + 1 + ⋯ + γ T − t r T ∣ S t ] , γ ∈ [ 0 , 1 ] V(S_t) = \mathbb{E}_{\tau \sim \pi}\left[r_t + \gamma r_{t+1} + \dots + \gamma^{T-t} r_T \mid S_t\right], \qquad \gamma \in [0, 1] V ( S t ) = E τ ∼ π [ r t + γ r t + 1 + ⋯ + γ T − t r T ∣ S t ] , γ ∈ [ 0 , 1 ]
V ( S t ) = E τ ∼ π [ G t ∣ S t ] ( G t = total discounted return ) V(S_t) = \mathbb{E}_{\tau \sim \pi}\left[G_t \mid S_t\right] \quad (G_t = \text{total discounted return}) V ( S t ) = E τ ∼ π [ G t ∣ S t ] ( G t = total discounted return )
To train our model V ϕ V_\phi V ϕ to converge to the ideal value G t G_t G t , we use gradient descent:
L t V F = ( V ϕ ( S t ) − G t ) 2 (the loss function) L_t^{VF} = \left(V_\phi(S_t) - G_t\right)^2 \quad \text{(the loss function)} L t V F = ( V ϕ ( S t ) − G t ) 2 (the loss function)
Now, to compute the quality of action that we took, let’s denote the quality of action = Q ( s t , a t ) = Q(s_t, a_t) = Q ( s t , a t ) .
A t = Q ( s t , a t ) − V ( S t ) A_t = Q(s_t, a_t) - V(S_t) A t = Q ( s t , a t ) − V ( S t )
Q ( s t , a t ) = r t + γ r t + 1 + ⋯ + γ T − t r T Q(s_t, a_t) = r_t + \gamma r_{t+1} + \dots + \gamma^{T-t} r_T Q ( s t , a t ) = r t + γ r t + 1 + ⋯ + γ T − t r T
Defining Q Q Q this way is the Monte-Carlo approach .
This has downsides →
We need to wait for the complete action to unfold.
This value will be trivial when we don’t have intermediate rewards.
Instead we can do,
Q ( s t , a t ) = r t + γ [ r t + 1 + γ r t + 2 + ⋯ + γ T − 2 r T ] Q(s_t, a_t) = r_t + \gamma\left[r_{t+1} + \gamma r_{t+2} + \dots + \gamma^{T-2} r_T\right] Q ( s t , a t ) = r t + γ [ r t + 1 + γ r t + 2 + ⋯ + γ T − 2 r T ]
Q ^ ( s t , a t ) = r t + γ V ϕ ( s t + 1 ) ← this is bootstrapping \hat{Q}(s_t, a_t) = r_t + \gamma V_\phi(s_{t+1}) \quad \leftarrow \text{this is bootstrapping} Q ^ ( s t , a t ) = r t + γ V ϕ ( s t + 1 ) ← this is bootstrapping
So,
A ^ t = r t + γ V ϕ ( s t + 1 ) − V ϕ ( s t ) \hat{A}_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t) A ^ t = r t + γ V ϕ ( s t + 1 ) − V ϕ ( s t )
In practice, things are let’s say different!
In theory, we define expectation over a very large sample size. But in practice, our sample size is not that large. Also sometimes we deal with noisy environments, rewards also comes from humans so we can’t expect them to be correct. This problem is Variance → can be fixed by giving less importance to rewards but that leads to a very conservative model that can’t discover the optimal solution. This over correction is called bias . And all this is called the Bias-Variance tradeoff .
Q ( s t , a t ) = r t + γ r t + 1 + ⋯ + γ T − 1 r T (high variance, low bias) Q(s_t, a_t) = r_t + \gamma r_{t+1} + \dots + \gamma^{T-1} r_T \quad \text{(high variance, low bias)} Q ( s t , a t ) = r t + γ r t + 1 + ⋯ + γ T − 1 r T (high variance, low bias)
Q ^ ( s t , a t ) = r t + γ V ϕ ( s t + 1 ) (low variance, high bias) \hat{Q}(s_t, a_t) = r_t + \gamma V_\phi(s_{t+1}) \quad \text{(low variance, high bias)} Q ^ ( s t , a t ) = r t + γ V ϕ ( s t + 1 ) (low variance, high bias)
Between these two ends of spectra, we can have:
Q ^ 2 ( s t , a t ) = r t + γ r t + 1 + γ 2 V ϕ ( s t + 2 ) \hat{Q}_2(s_t, a_t) = r_t + \gamma r_{t+1} + \gamma^2 V_\phi(s_{t+2}) Q ^ 2 ( s t , a t ) = r t + γ r t + 1 + γ 2 V ϕ ( s t + 2 )
Q ^ 3 ( s t , a t ) = r t + γ r t + 1 + γ 2 r t + 2 + γ 3 V ϕ ( s t + 3 ) \hat{Q}_3(s_t, a_t) = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \gamma^3 V_\phi(s_{t+3}) Q ^ 3 ( s t , a t ) = r t + γ r t + 1 + γ 2 r t + 2 + γ 3 V ϕ ( s t + 3 )
⋮ \vdots ⋮
The corresponding advantage estimates will be:
A ^ 1 = r t + γ V ϕ ( s t + 1 ) − V ( s t ) = δ t \hat{A}_1 = r_t + \gamma V_\phi(s_{t+1}) - V(s_t) = \delta_t A ^ 1 = r t + γ V ϕ ( s t + 1 ) − V ( s t ) = δ t
A ^ 2 = r t + γ r t + 1 + γ 2 V ϕ ( s t + 2 ) − V ( s t ) = δ t + 1 \hat{A}_2 = r_t + \gamma r_{t+1} + \gamma^2 V_\phi(s_{t+2}) - V(s_t) = \delta_{t+1} A ^ 2 = r t + γ r t + 1 + γ 2 V ϕ ( s t + 2 ) − V ( s t ) = δ t + 1
⋮ \vdots ⋮
A ^ T = r t + γ r t + 1 + ⋯ + γ T − 1 r T − V ( s T ) \hat{A}_T = r_t + \gamma r_{t+1} + \dots + \gamma^{T-1} r_T - V(s_T) A ^ T = r t + γ r t + 1 + ⋯ + γ T − 1 r T − V ( s T )
A ^ t \hat{A}_t A ^ t = exponentially weighted average (A ^ 1 … A ^ T \hat{A}_1 \dots \hat{A}_T A ^ 1 … A ^ T ) with weight λ \lambda λ .
A ^ t = ∑ k = 0 T − t ( γ λ ) k δ t + k (General advantage estimation) \hat{A}_t = \sum_{k=0}^{T-t} (\gamma\lambda)^k \, \delta_{t+k} \quad \text{(General advantage estimation)} A ^ t = ∑ k = 0 T − t ( γλ ) k δ t + k (General advantage estimation)
Initialize policy θ 0 \theta_0 θ 0 and value function ϕ 0 \phi_0 ϕ 0
for n = 0 , 1 , 2 , … n = 0, 1, 2, \dots n = 0 , 1 , 2 , …
collect set of trajectories { τ i } \{\tau_i\} { τ i } by executing policy π θ n \pi_{\theta_n} π θ n within the environment
compute discounted returns (G t G_t G t )
compute A t A_t A t based on V ϕ n V_{\phi_n} V ϕ n
update the policy based on ∇ θ L P G \nabla_\theta L^{PG} ∇ θ L PG
update value function based on ∇ ϕ L V F \nabla_\phi L^{VF} ∇ ϕ L V F
In supervised learning, it’s std. to have multiple epochs. PPO revisits the sets of trajectories multiple times. But this creates a problem: sampling the response is using policy parameters from n n n th model update (π θ n \pi_{\theta_n} π θ n ) but we are computing loss using new parameters θ n + epoch \theta_{n+\text{epoch}} θ n + epoch . This can break the training process as we are backpropagating the wrong gradient.
There’s a maths trick called Importance Sampling . It allows us to estimate expectations under one distribution p ( x ) p(x) p ( x ) from another distribution q ( x ) q(x) q ( x ) .
E x ∼ p [ g ( x ) ] = E x ∼ q [ g ( x ) ⋅ p ( x ) q ( x ) ⏟ Surrogate loss ] \mathbb{E}_{x \sim p}[g(x)] = \mathbb{E}_{x \sim q}\left[g(x) \cdot \underbrace{\frac{p(x)}{q(x)}}_{\text{Surrogate loss}}\right] E x ∼ p [ g ( x )] = E x ∼ q g ( x ) ⋅ Surrogate loss q ( x ) p ( x )
θ o l d \theta_{old} θ o l d : Sampled trajectories from π o l d \pi_{old} π o l d
θ n e w \theta_{new} θ n e w : Computed loss using π θ n e w \pi_{\theta_{new}} π θ n e w
L P G = ∑ t [ π θ n e w ( a t ∣ s t ) π θ o l d ( a t ∣ s t ) A t ] = ∑ t [ r ( θ ) ⋅ A t ] L^{PG} = \sum_t \left[\frac{\pi_{\theta_{new}}(a_t \mid s_t)}{\pi_{\theta_{old}}(a_t \mid s_t)} \, A_t\right] = \sum_t \left[r(\theta) \cdot A_t\right] L PG = ∑ t [ π θ o l d ( a t ∣ s t ) π θ n e w ( a t ∣ s t ) A t ] = ∑ t [ r ( θ ) ⋅ A t ]
π θ n e w ( a t ∣ s t ) π θ o l d ( a t ∣ s t ) \dfrac{\pi_{\theta_{new}}(a_t \mid s_t)}{\pi_{\theta_{old}}(a_t \mid s_t)} π θ o l d ( a t ∣ s t ) π θ n e w ( a t ∣ s t ) reflects how much the model changes between two updates.
PPO clips this between 1 − ϵ 1-\epsilon 1 − ϵ and 1 + ϵ 1+\epsilon 1 + ϵ , where ϵ ← \epsilon \leftarrow ϵ ← hyperparameter.
L C L I P = ∑ t [ min ( r t ( θ ) , clip ( 1 − ϵ , r t ( θ ) , 1 + ϵ ) ) ] ⋅ A t L^{CLIP} = \sum_t \left[\min\left(r_t(\theta), \; \text{clip}\left(1-\epsilon, \; r_t(\theta), \; 1+\epsilon\right)\right)\right] \cdot A_t L C L I P = ∑ t [ min ( r t ( θ ) , clip ( 1 − ϵ , r t ( θ ) , 1 + ϵ ) ) ] ⋅ A t
We can also penalise the difference in policy distribution by substrating the KL divergence between both the policies
L PPO ( θ ) = E s , a ∼ π θ [ min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) ] − β D KL ( π ref ∥ π θ ) \mathcal{L}_{\text{PPO}}(\theta) = \mathbb{E}_{s,a \sim \pi_\theta}\left[\min\left(r_t(\theta)\hat{A}_t, \; \text{clip}\left(r_t(\theta), 1-\epsilon, 1+\epsilon\right)\hat{A}_t\right)\right] - \beta D_{\text{KL}}\left(\pi_{\text{ref}} \,\|\, \pi_\theta\right) L PPO ( θ ) = E s , a ∼ π θ [ min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) ] − β D KL ( π ref ∥ π θ )
KL ( P ∥ Q ) = ∫ log ( p ( x ) q ( x ) ) p ( x ) d x = E x ∼ p [ log ( p ( x ) q ( x ) ) ] \text{KL}(P \,\|\, Q) = \int \log\left(\frac{p(x)}{q(x)}\right) p(x) \, dx = \mathbb{E}_{x \sim p}\left[\log\left(\frac{p(x)}{q(x)}\right)\right] KL ( P ∥ Q ) = ∫ log ( q ( x ) p ( x ) ) p ( x ) d x = E x ∼ p [ log ( q ( x ) p ( x ) ) ]
I found this very helpful video for understanding PPO
GRPO eliminates the need of having a value function!
Instead of a value function, it uses the average reward of multiple sampled outputs, produced in response to the same question as baseline.
For each question q q q , GRPO samples a group of outputs { o 1 , o 2 , … , o G } \{o_1, o_2, \dots, o_G\} { o 1 , o 2 , … , o G } from the old policy π θ o l d \pi_{\theta_{old}} π θ o l d and then optimizes the policy model by maximizing the following objective:
J G R P O ( θ ) = E [ q ∼ P ( Q ) , { o i } i = 1 G ∼ π θ o l d ( O ∣ q ) ] 1 G ∑ i = 1 G 1 ∣ o i ∣ ∑ t = 1 ∣ o i ∣ { min [ π θ ( o i , t ∣ q , o i , < t ) π θ o l d ( o i , t ∣ q , o i , < t ) A ^ i , t , clip ( π θ ( o i , t ∣ q , o i , < t ) π θ o l d ( o i , t ∣ q , o i , < t ) , 1 − ε , 1 + ε ) A ^ i , t ] − β D K L [ π θ ∥ π r e f ] } , \mathcal{J}_{GRPO}(\theta) = \mathbb{E}_{\left[q \sim P(Q),\, \{o_i\}_{i=1}^{G} \sim \pi_{\theta_{old}}(O|q)\right]}
\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}
\left\{
\min\left[
\frac{\pi_\theta(o_{i,t}\mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}\mid q, o_{i,<t})}\hat{A}_{i,t},\;
\operatorname{clip}\left(
\frac{\pi_\theta(o_{i,t}\mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}\mid q, o_{i,<t})},\,
1-\varepsilon,\, 1+\varepsilon
\right)\hat{A}_{i,t}
\right]
- \beta\,\mathbb{D}_{KL}\left[\pi_\theta \,\|\, \pi_{ref}\right]
\right\}, J GRPO ( θ ) = E [ q ∼ P ( Q ) , { o i } i = 1 G ∼ π θ o l d ( O ∣ q ) ] G 1 i = 1 ∑ G ∣ o i ∣ 1 t = 1 ∑ ∣ o i ∣ { min [ π θ o l d ( o i , t ∣ q , o i , < t ) π θ ( o i , t ∣ q , o i , < t ) A ^ i , t , clip ( π θ o l d ( o i , t ∣ q , o i , < t ) π θ ( o i , t ∣ q , o i , < t ) , 1 − ε , 1 + ε ) A ^ i , t ] − β D K L [ π θ ∥ π re f ] } ,
where ε \varepsilon ε and β \beta β are hyperparameters.
Also note that, instead of adding KL penalty in the reward, GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss, avoiding complicating the calculation of Advantage
Algorithm: Iterative Group Relative Policy Optimization
Input: initial policy model π_θinit; reward models r_φ; task prompts D;
hyperparameters ε, β, μ
policy model π_θ ← π_θinit
for iteration = 1, …, I do
reference model π_ref ← π_θ
for step = 1, …, M do
Sample a batch D_b from D
Update the old policy model π_θold ← π_θ
Sample G outputs {o_i}, i=1..G ~ π_θold(· | q) for each question q ∈ D_b
Compute rewards {r_i}, i=1..G for each sampled output o_i by running r_φ
Compute Â_{i,t} for the t-th token of o_i through group relative
advantage estimation
for GRPO iteration = 1, …, μ do
Update the policy model π_θ by maximizing the GRPO objective
(Equation 21)
Update r_φ through continuous training using a replay mechanism
Output: π_θ
Outcome supervision provides the normalized reward at the end of each output o i o_i o i and sets the advantages A ^ i , t \hat{A}_{i,t} A ^ i , t of all tokens in the output as the normalized reward, i.e.,
A ^ i , t = r ~ i = r i − mean ( r ) std ( r ) , \hat{A}_{i,t} = \widetilde{r}_i = \frac{r_i - \operatorname{mean}(\mathbf{r})}{\operatorname{std}(\mathbf{r})}, A ^ i , t = r i = std ( r ) r i − mean ( r ) ,
and then optimizes the policy by maximizing the objective defined in the reward function equation.