Reinforcement Learning (RL) has become one of the core ingredients of modern Large Language Model (LLM) post-training. While pre-training teaches a model to predict the next token from massive text corpora, post-training aims to shape its behavior: following instructions, being helpful, remaining harmless, and ultimately aligning its responses with human preferences. RL offers a natural framework for this problem, allowing the model to optimize goals that are difficult to express through supervised learning alone. In this post, we gonna walk through how RL has been adapted to LLMs post-training, covering the main algorithms used in modern language models and the ideas that motivated their development.
To understand how RL is applied to LLMs, it is useful to reinterpret autoregressive text generation as a sequential decision-making problem (Ranzato et al. 2015). Instead of viewing the model as merely predicting the next token, we consider each generated token as an action taken by an agent interacting with its environment. This simple change in perspective allows us to formulate language generation as a Markov Decision Process (MDP), making the entire RL toolbox applicable. Let’s take a look at the example below:
$$\begin{aligned} s_0 &: \texttt{"What national team has more World Cups?"} \\ a_0 &: \texttt{"Brazil"} \\ s_1 &: \texttt{"What national team has more World Cups? Brazil"} \\ a_1 &: \texttt{"has"} \\ s_2 &: \texttt{"What national team has more World Cups? Brazil has"} \\ a_2 &: \texttt{"5"} \\ s_3 &: \texttt{"What national team has more World Cups? Brazil has 5"} \\ a_3 &: \texttt{"World Cups."} \\ s_4 &: \texttt{"What national team has more World Cups? Brazil has 5 World Cups."} \end{aligned}$$Given an initial state $s_0$ (the user’s prompt), the policy sequentially generates tokens according to $a \sim \pi(\cdot \mid s_t)$. The episode $\tau = (s_0, a_0, s_1, \dots, s_t)$ terminates when the model produces an end-of-sequence token or reaches a predefined length limit, thereby completing the response. Due to the low granularity of token-level actions and the inability to measure their individual contribution to the final answer, the response is typically evaluated only after the episode ends, resulting in a sparse reward signal. Learning from such delayed feedback is precisely the setting addressed by Reinforcement Learning from Human Feedback (RLHF). Broadly speaking, there are two ways of obtaining rewards from humans:
Human feedback: humans directly evaluate model outputs, for instance by comparing two responses or assigning quality scores. While this provides the most faithful supervision, it is also expensive, slow, and inherently noisy, since human preferences can be ambiguous or even contradictory.
Learned reward: instead of querying humans every time, we train a reward model on previously collected human feedback. This model predicts a scalar reward for each completion, making RL training significantly cheaper. However, it inevitably inherits both the inconsistencies of human annotations and the approximation errors introduced by the learning process.
Suppose we are given a reward function that evaluates a complete response generated by a language model. Our goal is to update the policy so that high-quality responses become more likely over time. Formally, RL seeks to maximize the expected return over trajectories,
$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ R(\tau) \right].$$Here, a trajectory $\tau$ denotes a complete interaction generated by the policy, while $R(\tau)$ is the return assigned to that trajectory. Since the policy induces a probability distribution over trajectories, the objective can be written as
$$J(\theta) = \int \pi_\theta(\tau) R(\tau) \,d\tau.$$To optimize this objective, we differentiate it with respect to the policy parameters,
$$\nabla_\theta J(\theta) = \int R(\tau) \nabla_\theta \pi_\theta(\tau) \,d\tau.$$At first glance, this expression seems amenable to Monte Carlo estimation by sampling trajectories from the current policy. The problem, however, is that the integrand contains $\nabla_\theta \pi_\theta(\tau)$, which is not itself a probability distribution. Consequently, the integral cannot be estimated directly from sampled trajectories. To overcome this issue, we can employ the log-derivative trick,
$$\nabla_\theta \pi_\theta(\tau) = \pi_\theta(\tau) \nabla_\theta \log \pi_\theta(\tau),$$which allows us to rewrite the gradient as
$$\nabla_\theta J(\theta) = \int \pi_\theta(\tau) \left[ R(\tau) \nabla_\theta \log \pi_\theta(\tau) \right] d\tau.$$The integrand is now expressed as the product of a probability distribution and a function, allowing us to rewrite it as an expectation,
$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta} \left[ R(\tau) \nabla_\theta \log \pi_\theta(\tau) \right].$$Finally, this expectation can be approximated using Monte Carlo samples, which corresponds to the classical REINFORCE estimator introduced by Williams (1992)
$$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} R(\tau_i) \nabla_\theta \log \pi_\theta(\tau_i),$$For LLMs, a trajectory corresponds to a complete generated response. Since the policy generates one token at a time, the probability of a trajectory factorizes autoregressively as
$$\log \pi_\theta(\tau) = \sum_{t=0}^{T-1} \log \pi_\theta(a_t \mid s_t),$$yielding
$$\nabla_\theta J(\theta) = \mathbb{E} \left[ R(\tau) \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t\mid s_t) \right].$$However, REINFORCE is known to suffer from high variance, which is mitigated by a baseline as explained in my previous post. This baseline can be, for example, an exponential moving average (EMA) of previous returns, yielding:
$$\nabla_\theta J(\theta) = \mathbb{E} \left[ (R(\tau) - b) \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t\mid s_t) \right].$$Although REINFORCE provides an unbiased estimator of the policy gradient, it is well known to suffer from high variance, often leading to unstable optimization. A natural question therefore arises: can we reduce this variance without changing the expected gradient? Fortunately, the answer is yes. By subtracting any baseline $b(s)$ that does not depend on the sampled action, we obtain
$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta} \left[ \left(R(\tau)-b(s)\right) \nabla_\theta \log \pi_\theta(\tau) \right].$$Since the expected gradient of the baseline term is zero, this estimator remains unbiased while often exhibiting significantly lower variance. But if a simple baseline already helps, why not learn the best possible one? This is precisely the motivation behind the actor-critic family of algorithms, in which the baseline is replaced by a learned state-value function $V_\phi(s)$.
Imagine the following scenario: you are learning a task, trying to map every possible state to an action, while a colleague is learning from the very same experience, helping you estimate how good it is to be in each state. That is essentially the actor-critic architecture. You (the actor) learn a policy $\pi_\theta(a_t\mid s_t)$, while your colleague (the critic) learns a value function $V_\phi(s_t)$ that estimates the expected future return starting from the current state. Sounds great, doesn’t it? Then why not use your colleague’s estimate as the baseline $b$ introduced in REINFORCE?
Instead of relying on a baseline, we now compare the observed return against what the critic expected to happen. This gives rise to the so-called advantage function,
$$A_t = R_t - V_\phi(s_t),$$which measures whether an action performed better or worse than expected. The idea is simple: if $A_t > 0$, the outcome was a positive surprise and we should make that action more likely in the future. Conversely, if $A_t < 0$, the action performed worse than expected and its probability should be reduced. Plugging this expression into the policy gradient objective yields
$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t\mid s_t) A_t \right].$$So, problem solved? Not quite. Although actor-critic methods dramatically reduce the variance of policy gradients, the policy can still change too aggressively after a single update, occasionally causing catastrophic drops in performance. The remaining question is therefore: how can we make policy updates more conservative? One elegant answer is provided by Proximal Policy Optimization (PPO) (Schulman et al. 2017), which clips the policy objective to prevent excessively large updates.
So, first of all, we need to establish a measure of how much the policy is changing. PPO does this through the probability ratio
$$r_t(\theta) = \frac{\pi_\theta(a_t\mid s_t)} {\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)},$$where $\pi_{\theta_{\mathrm{old}}}$ is a frozen copy of the policy used to collect the current batch of trajectories. Intuitively, $r_t>1$ means that the new policy assigns a higher probability to the sampled action, whereas $r_t<1$ means the opposite. Combining this ratio with the advantage function yields the clipped PPO objective
$$L_{\mathrm{PPO}}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta)A_t,\, \mathrm{clip} \left( r_t(\theta), \underbrace{1-\epsilon}_{\text{lower bound}}, \underbrace{1+\epsilon}_{\text{upper bound}} \right) A_t \right) \right],$$where $\epsilon$ is a small hyperparameter (typically $0.1$ or $0.2$) that controls how conservative policy updates are. The clipping operation does not explicitly prevent the policy from leaving the interval $[1-\epsilon,,1+\epsilon]$. Instead, once the probability ratio exceeds these bounds, the objective no longer rewards further changes in that direction. As a result, excessively large policy updates become progressively less attractive, allowing PPO to safely perform multiple optimization epochs over the same batch while keeping successive policies reasonably close to one another.
Now, the cherry on top! In RL post-training for LLMs, an additional Kullback–Leibler (KL) penalty is typically introduced to make updates even more conservative (yeah, layers and layers of conservatism). Usually, we start from a reference policy obtained through supervised fine-tuning (SFT) on task-specific demonstrations. RL then acts as an alignment step, encouraging the model to generate responses that better reflect human preferences while preserving the linguistic capabilities acquired during pre-training and SFT. This is achieved by augmenting the PPO objective with a KL penalty,
$$L(\theta) =L_{\mathrm{PPO}}(\theta)- \beta\, D_{\mathrm{KL}} \!\left( \pi_\theta \,\|\, \pi_{\mathrm{SFT}} \right),$$where $\pi_{\mathrm{SFT}}$ denotes the frozen SFT policy and $\beta$ controls the strength of penalties for deviations from this reference model.
I often ask LLMs to formalize ideas. The reason is simple: it is usually much easier to recognize a good answer than to write one from scratch. This simple observation is one of the main motivations behind RLHF. Instead of asking annotators to produce perfect demonstrations, we can simply ask them which of two responses they prefer. Not only is this easier for humans, but it also provides a much richer supervision signal than the binary notion of right or wrong typically used in supervised learning.
Of course, this raises a natural question: how do we transform pairwise preferences into something a learning algorithm can optimize? The classical answer is provided by the Bradley–Terry model (Bradley and Terry 1952), which assumes that the probability of preferring one response over another depends on their underlying latent rewards. Let’s take, as an example, two completions $y_w$ and $y_l$
$$P(y_w \succ y_l)= \sigma\!\left( s(y_w)-s(y_l) \right),$$where
$$\sigma(x)=\frac{1}{1+e^{-x}}.$$In this example, we have two completions, $y_w$ and $y_l$, where $w$ stands for winner, i.e., the preferred response, and $l$ denotes the loser. The Bradley–Terry model first assigns a latent score $s_\phi(x,y)$ to each completion. We then compute their score difference
$$\Delta s = s_\phi(x,y_w)- s_\phi(x,y_l),$$and map it through a sigmoid function $\sigma$. Large positive values are mapped close to $1$, large negative values close to $0$, while a zero difference is mapped exactly to $0.5$, as illustrated in Figure below.
Voilà, we now have a probability associated with each pairwise preference,
$$P(y_w \succ y_l \mid x) = \sigma \!\left(s_\phi(x,y_w)- s_\phi(x,y_l)\right).$$The next step is to make this score differentiable by maximizing the likelihood of the observed human preferences—or, equivalently, minimizing the negative log-likelihood,
$$\mathcal{L}_{\mathrm{BT}}(\phi) = -\log \sigma \!\left(s_\phi(x,y_w)- s_\phi(x,y_l) \right).$$The remaining question, though, is: how do we obtain the scores associated with $y_w$ and $y_l$? This is precisely the idea behind Direct Preference Optimization (DPO) (Rafailov et al. 2023). The intuition is remarkably simple: why should we train a separate scoring model if the policy already assigns a probability to every completion? Intuitively, the more likely the policy considers a completion, the more confident it is about generating it. A natural first attempt is therefore to define the score directly as the log-probability assigned by the policy,
$$s_\phi(x,y) = \log \pi_\theta(y \mid x) = \sum_{t=0}^{T-1} \log \pi_\theta\!\left(y_t \mid x, y_{1:t-1}\right) $$where the second equality follows from the autoregressive factorization of language models. Unfortunately, this naive formulation suffers from an important problem. The policy probabilities alone do not distinguish whether a completion is intrinsically good or simply reflects the language distribution learned during pre-training. In other words, we would like to compare the current policy against the SFT policy rather than using its raw probabilities, such as
$$\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\left[\log\sigma\left(\beta\log \frac{\pi_\theta(y_w\mid x)} {\pi_{\mathrm{SFT}}(y_w\mid x)}- \beta \log \frac{\pi_\theta(y_l\mid x)} {\pi_{\mathrm{SFT}}(y_l\mid x)} \right) \right].$$where $\beta$ controls how strongly the optimized policy is anchored to the SFT policy. Rather than maximizing the raw probability of the preferred completion, DPO increases its probability relative to the reference model, while doing the opposite for the rejected completion. In other words, the reward model, the critic, and the whole PPO optimization loop disappear: preference learning is reduced to a simple classification objective over pairs of responses.
To conclude our journey through RL post-training methods, let us look at another popular algorithm that reduces the computational overhead introduced by actor-critic methods: Group Relative Policy Optimization (GRPO) (Shao et al. 2024). The motivation, again, is straighforward. Instead of estimating the value of a state through a learned critic, why not generate several completions for the same prompt and use the group itself as a reference? In other words, if a completion performs better than the average completion generated for the same prompt, we should encourage it; otherwise, we should discourage it.
Formally, let $(y_1,\ldots,y_G)$ denote a group of $G$ completions sampled from the current policy, with corresponding rewards $(R_1,\ldots,R_G)$. GRPO defines the advantage of the $i$-th completion as
$$A_i = \frac{R_i-\bar{R}} {\sigma_R}, \qquad \bar{R} = \frac{1}{G} \sum_{j=1}^{G} R_j,$$where $\sigma_R$ denotes the standard deviation of the rewards within the group. This simple idea eliminates the need for a learned critic. Instead of comparing a completion against an estimated value function, GRPO compares it against the average quality of its peers. This significantly reduces the computational cost of training, which partly explains its popularity in recent reasoning models.
The price to pay, however, is theoretical. Unlike the baseline used in REINFORCE, the group average depends on the sampled actions themselves, meaning it is no longer independent of the current sample. Consequently, the resulting policy-gradient estimator is biased, although in practice this bias has proven small enough to be largely outweighed by the computational savings. The optimization objective therefore remains essentially identical to PPO, differing only in how the advantage function is estimated.
Bradley, Ralph Allan, and Milton E. Terry. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39:324.
Williams, Ronald J. 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning 8 (3–4): 229–256. https://doi.org/10.1007/BF00992696.
Ranzato, Marc’Aurelio, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence Level Training with Recurrent Neural Networks. arXiv:1511.06732.
Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347.
Rafailov, Rafael, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. NeurIPS 2023.
Shao, Zhihong, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.