Modern PPO for Multi-Turn LLM Agents: An Implementation Guide

September 6, 2026

This article is for engineers and researchers implementing proximal policy optimization (PPO) for autoregressive language-model agents whose trajectories interleave model tokens, tool results, environment observations, and padding. It begins with ordinary episodic PPO, then develops the tensor-level machinery required for multi-turn training: reward placement, masking, value bootstrapping, loss aggregation, and correction for asynchronous rollouts. Generalized advantage estimation (GAE) receives particular attention because its recursion follows model decisions rather than raw token positions.

Correctness depends on preserving the Markov decision process (MDP) induced by the model’s decision points. Only tokens sampled from the model are actions. Tool results and environment observations become part of the next state but do not advance the GAE recursion. With this interpretation fixed, we can define response masks, termination and truncation boundaries, policy roles, and loss denominators precisely. The remaining techniques are optional stabilizers for separate failure modes, not parts of one mandatory recipe.

1. From ordinary PPO to multi-turn agent PPO

1.1 The minimal ordinary PPO loop

In ordinary episodic PPO, let the environment time step be $h=0,1,\ldots,H-1$. In state $s_h$, the policy samples action $a_h$, and the environment returns reward $r_h$ and next state $s_{h+1}$. A trajectory is written as the ordered sequence

where $H$ is the number of transitions and $s_H$ is the final state. We use the following notation:

• $\pi_\theta$: the current policy, parameterized by $\theta$ and being optimized;
• $\mu$: the behavior policy that actually samples the trajectory;
• $\pi_{\mathrm{old}}$: the proximal policy frozen before the current update and used in the PPO ratio and clipping operation. In synchronous PPO, it usually also generates the current batch; in asynchronous PPO, the behavior policy may differ;
• $V_\phi$: the critic parameterized by $\phi$. GAE uses a frozen pre-update critic with parameters $\phi_{\mathrm{old}}$; we write its prediction as $\widehat V_h:=V_{\phi_{\mathrm{old}}}(s_h)$;
• $d_h\in{0,1}$: whether true termination occurs after transition $h$;
• $\gamma\in[0,1]$: the reward discount;
• $\lambda\in[0,1]$: the GAE trace parameter.

Ordinary synchronous PPO generally has $\mu=\pi_{\mathrm{old}}$. Keeping the two symbols distinct from the outset lets us use the same ratio definitions in the later discussion of asynchronous rollouts.

Ordinary GAE first computes the temporal-difference (TD) residual

then recurses from right to left:

$A_h$ is the advantage used by the actor. If the critic uses the same GAE parameters, its regression target is

Here $\operatorname{stopgrad}(\cdot)$ means that the quantity in parentheses is used only as a fixed training target and that backpropagation does not pass through the operator.

The PPO probability ratio and clipped objective are [1][2]

where $\epsilon_{\mathrm{clip}}\in(0,1)$ is the policy-ratio clipping radius. The actor maximizes $L_h^{\mathrm{PPO}}$; when this article writes a loss to be minimized, it uses $\ell_h^{\mathrm{PPO}}:=-L_h^{\mathrm{PPO}}$.

1.2 Mapping LLM tokens to PPO states and actions

For an autoregressive LLM, each sampled next token is an action. For sample $i$ in a batch, let:

• $B$: the number of sequences in the batch, with $i\in{1,\ldots,B}$;
• $N_i$: the total tensor length of sequence $i$, including padding;
• $c_{i,t}\in{0,1}$: the attention-validity mask at tensor position $t$;
• $x_{i,t}$: the token at raw token position $t\in{1,\ldots,N_i}$;
• $s_{i,t}$: the actual model input state for predicting position $t$, consisting of the attention-visible prefix selected by $c_i$ together with its attention and positional information; padding is not part of this state;
• $a_{i,t}:=x_{i,t}$: the corresponding PPO action when position $t$ is generated by the model.

Therefore,

Single-turn language-model PPO often has one contiguous response. A multi-turn agent sequence instead interleaves model output with tool results, environment observations, system messages, and padding. Only model output contains policy actions. The intervening content becomes part of the state seen at the next model decision. We therefore define an induced MDP over model decision points and use a response mask to run GAE on that compressed timeline.

2. Interleaved trajectories: masks, spans, rewards, and termination semantics

2.1 Environment steps and raw token positions are different timelines

At the environment level, interaction step $k$ starts from state $S_{i,k}$. The model generates content $Z_{i,k}$, from which the system parses an executable action $A_{i,k}$. The environment then returns reward $R_{i,k}$ and the next observation $O_{i,k+1}$. We write the ordered interaction record as

where $k\in{1,\ldots,K_i}$ and $K_i$ is the number of environment interactions. The observation $O_{i,k+1}$, together with any other environment update, contributes to the next environment state $S_{i,k+1}$. Uppercase letters distinguish this environment-level description from token-level $s_{i,t}$ and $a_{i,t}$.

Serialization produces

The environment step $k$, raw token position $t$, and compressed model-token decision step $j$ introduced below are not the same index. One environment step may contain many model tokens, and many observation tokens may appear between two environment actions. Model-output spans use a separate index $p$.

2.2 Response masks and model-output spans

Using the behavior policy $\mu$ defined above, define the response mask as [3][4]

By construction, $m_{i,t}\le c_{i,t}$: a padding position can never be a policy action.

The mask is 1 for model-generated tokens and 0 for prompts, tool results, environment observations, and padding. The discussion first considers samples containing at least one model token; samples with no model tokens must be excluded from any aggregation whose denominator counts valid model tokens. Fix one sample and omit subscript $i$, writing $N:=N_i$ and $m_t:=m_{i,t}$. Suppose the sequence has $M\ge1$ contiguous model-output spans. For $p\in{1,\ldots,M}$, let $b_p,e_p$ be the starting and ending token positions of span $p$, respectively, and define the discrete closed interval

The endpoints satisfy

The set of valid model tokens is

A typical mask has the form

The mask has two uses. First, it excludes prompt, observation, and padding tokens from actor and critic losses and from token-level statistics. Second, it prevents those positions from advancing the GAE recursion. Loss masking alone does not provide the second behavior. Under the decision-point MDP used in this article, $\gamma$ and $\lambda$ are applied once per valid model token, regardless of how many observation tokens or how much wall-clock time separates two decisions. Other timescale conventions are possible, but they define different estimators.

2.3 From environment rewards to token rewards

Let $u_{i,t}$ be the reward-alignment tensor passed to token-level GAE. Its position is a bookkeeping choice and need not match the physical time at which the system receives the reward. For example, feedback returned after a tool executes can be assigned to the final token of the preceding tool call. This treats the feedback as the transition reward from that model action to the next model decision state. If sample $i$ has $J_i:=\sum_t m_{i,t}>0$ model tokens and the task provides only a trajectory-level score $R(\tau_i)$, define its final valid model position as

and place the terminal reward there:

The last valid model token must be used rather than the end of the tensor, because the tensor may end with an observation or padding.

If model-output span $p$ receives turn feedback $g_p$, a common alignment adds it to the end of that span. Fixing the current sample and omitting subscript $i$ gives

If Kullback-Leibler (KL) reward shaping against a reference policy is enabled, apply it before computing GAE in Section 3. Let $\pi_{\mathrm{ref}}$ be the frozen reference policy and define the sampled rollout-time log-ratio as

and let

where $\beta_{\mathrm{KL},r}\ge0$ and $k_{i,t}^{\mathrm{rollout}}$ may be set to 0 when $m_{i,t}=0$. When this term is enabled, every $u$ in Section 3 refers to $u^{\mathrm{total}}$; otherwise, it refers to the original task reward. A sampled log-ratio may be negative. Its expectation over actions sampled from $\mu$ is $D_{\mathrm{KL}}(\mu\Vert\pi_{\mathrm{ref}})$.

Turn-level reward and a turn-level advantage estimator must be distinguished. The former specifies only where a reward is produced; the latter specifies whether values, TD residuals, and advantages recurse by turn. Placing a reward at the end of a turn does not automatically convert token-level GAE into turn-level GAE. Explicit turn-level credit assignment requires an independently defined turn-level value or advantage [6][7][8].

2.4 True termination vs time-limit/context-length truncation

True termination means that the task has ended and the bootstrap value beyond the boundary is 0. Time-limit or context-length truncation means only that sampling was cut off. If the underlying task could continue, the estimator should bootstrap from the boundary state rather than treat the cutoff as terminal [2][5].

Below, $d_{i,j}\in{0,1}$ indicates whether sample $i$ truly terminates after its $j$th valid model-token transition. On normal completion, the final step generally has $d_{i,J_i}=1$. Under pure truncation, the final indicator is $d_{i,J_i}=0$ and an additional boundary value is supplied. For nonterminal internal valid steps $j<J_i$, $d_{i,j}=0$.

3. From standard GAE to masked model-token GAE

3.1 The compressed model-token decision timeline

For sample $i$, write the raw positions whose mask equals 1 in ascending order as

where

Here $j\in{1,\ldots,J_i}$ is the compressed model-token decision step, whereas $q_{i,j}$ is its raw token position. To simplify the notation, fix sample $i$ and omit its subscript throughout this section. GAE uses the frozen pre-update critic, so define

For $j<J$, the next policy action is at $q_{j+1}$, so the TD residual for masked model-token GAE is

Define the successor value for the final valid position as

where $s_{\mathrm{boundary}}$ is the successor state that the critic should evaluate after sampling stops if the task continues. It should contain any environment update or observation that has already occurred and become visible after the final valid model token; it cannot simply reuse the pre-action state $s_{q_J}$. If the system cannot construct this state, it cannot claim to perform correct truncation bootstrapping. Let $A_{J+1}:=0$. The unified recursion is then

where $\widehat V_{j+1}^{\ast}=\widehat V_{j+1}$ for $j<J$, while $\widehat V_{J+1}^{\ast}=\widehat V_{J+1}^{\mathrm{boot}}$. The factor $(1-d_j)$ is redundant with the zero terminal bootstrap in this notation but makes the termination semantics explicit. Finally, write $A_j$ back to raw token position $q_j$; positions with mask 0 produce no actor or critic target.

For example, if the raw mask is $(1,0,1)$, the valid positions are $q_1=1,q_2=3$, and position 2 is a tool result. Masked model-token GAE directly computes

The tool position itself produces no advantage and does not introduce another factor of $\gamma\lambda$. Its content has nevertheless entered the next decision state $s_{q_2}$, so it still affects $\widehat V_2$. The position is skipped only by the recursion clock; the model does not lose access to the observation.

Related implementations scan the raw response tensor from right to left but leave the next value and GAE trace unchanged when the response mask is 0. With their default zero-value initialization at the right boundary, this is mathematically equivalent to treating the final boundary in the equations above as terminal [3][4]. Observation and tool spans therefore consume no $\gamma\lambda$ factors, while every model-generated token consumes one step. This is masked model-token GAE, not turn-level GAE. If the training data includes time-limit or context-length truncation, a mask alone is insufficient: the estimator must also receive the boundary value and use it to initialize the final bootstrap. Otherwise, the actual computation still assigns a zero value to the truncated boundary.

3.2 Why a 20-token decay scale is not a hard horizon

In the GAE expansion, the weight of a future TD residual at distance $d\in\mathbb N_0$ valid model-token steps is

When $\gamma=1,\lambda=0.95$, $w(20)\approx0.358$, $w(50)\approx0.077$, and $w(100)\approx0.0059$. The commonly used expression

describes the decay scale of the TD-residual mixture, not a hard credit horizon; it is finite only when $\gamma\lambda<1$. Every residual contains the critic bootstrap for the next state, so a well-trained critic can compress information about more distant futures into adjacent values.

More concretely, consider a normally terminating trajectory in which only the final position $q_J$ receives terminal reward $R$ and $\gamma=1$. There are $D:=J-j$ valid steps from $q_j$ to the end. The recursion expands to

The term $\lambda^DR$ is the direct Monte Carlo component of the terminal reward, while the intermediate value terms are bootstrap components. A smaller $\lambda$ therefore relies more heavily on the critic rather than deleting long-range information altogether.

3.3 Optional extension: different traces for actor advantages and critic targets

Decoupled-GAE, introduced in VC-PPO [22] and adopted by VAPO [9], assigns different trace parameters to the actor and the critic. To prevent the critic target from moving with $\phi$ during optimization, first compute frozen value predictions using the pre-update parameters $\phi_{\mathrm{old}}$:

Every $V$ in the following GAE expressions refers to these frozen values $\widehat V$. Define

Standard PPO can set $A_j^{\mathrm{critic}}=A_j^{\mathrm{actor}}=A_j$; only Decoupled-GAE uses two traces. A typical choice is $\lambda_{\mathrm{critic}}\ge\lambda_{\mathrm{actor}}$: the critic target retains longer-return information, while the actor uses a shorter trace to control variance. Length-Adaptive GAE instead ties the actor trace to the sample’s valid length, choosing the GAE coefficients so that their geometric sum scales with $J_i$ [9]:

where $\alpha>0$ is a hyperparameter controlling the overall bias-variance trade-off (VAPO derives this by setting the trace scale $\sum_{t\ge0}\lambda_i^t\approx1/(1-\lambda_i)=\alpha J_i$). It remains a token-level estimator; only $\lambda_i$ differs across lengths.

3.4 Advantage whitening

Let $A_{i,j}^{\mathrm{actor}}$ denote the actor advantage at compressed model-token decision step $j$ in sample $i$; when written back to the raw tensor, it occupies position $q_{i,j}$. Let the total number of valid tokens in the batch be

Every $\sum_{i,j}$ below denotes $\sum_{i=1}^{B}\sum_{j=1}^{J_i}$. The mean, variance, and whitened advantage are [3]

where $\varepsilon_{\mathrm{var}}>0$ is a numerical-stability constant. Whitening stabilizes the actor’s advantage scale but must not change the critic target $G_{i,j}^{\mathrm{critic}}$.

Write $\widehat A_{i,j}^{\mathrm{policy}}$ for the final advantage passed to the PPO actor loss. With whitening enabled, set $\widehat A_{i,j}^{\mathrm{policy}}:=\widetilde A_{i,j}$; otherwise set $\widehat A_{i,j}^{\mathrm{policy}}:=A_{i,j}^{\mathrm{actor}}$.

4. Actor, critic, and regularization objectives

4.1 Token-level PPO-Clip

In synchronous PPO, the behavior policy is generally the proximal policy, so $\mu=\pi_{\mathrm{old}}$. For valid model token $(i,j)$, define

For brevity, omit subscripts $(i,j)$ below. The token loss to minimize is

Clip-Higher uses the asymmetric interval $[1-\epsilon_l,1+\epsilon_h]$, where $0<\epsilon_l<\epsilon_h$, permitting larger probability-ratio increases for positive-advantage tokens [10]. Dual-Clip adds a second upper bound for negative-advantage samples [11]:

where $\ell^{\mathrm{asym}}$ is the loss obtained by replacing the ordinary clipping interval with the asymmetric interval, and $c_{\mathrm{dual}}>1$ is the additional constraint on the negative-advantage branch. Both are optional stabilization mechanisms, not required components of multi-turn agent PPO. The quantity defined here is still a single-token loss; aggregating it into $\mathcal L_{\mathrm{policy}}$ requires choosing the token mean or sequence mean defined in Section 5.

4.2 Critic regression and value clipping

Restore the sample index by writing $\widehat V_{i,j}:=V_{\phi_{\mathrm{old}}}(s_{i,q_{i,j}})$. Let $V_{\mathrm{old},i,j}:=\widehat V_{i,j}$ be the frozen value prediction from before the current update, $V_{\phi,i,j}:=V_\phi(s_{i,q_{i,j}})$ the current critic prediction, and $\epsilon_V>0$ the value-clipping radius. Define

The clipped value loss is

Taking the maximum of the two terms means that, once an unclipped update has moved beyond the permitted range, clipping the prediction cannot artificially produce a smaller loss. Value clipping, reward scaling, and advantage normalization all substantially affect PPO results [3][12].

4.3 Actor-side KL regularization

Section 2.3 defined KL reward shaping, which must be applied before GAE. Another option is to add KL directly to the current actor loss. Let $\pi_{\mathrm{ref}}$ be the frozen reference policy and define the token mean of the full categorical KL as

The actor objective can include $\beta_{\mathrm{KL},L}\mathcal L_{\mathrm{KL}}$, where $\beta_{\mathrm{KL},L}\ge0$. Computing the full categorical KL requires summing over the vocabulary. An implementation may instead use an approximate estimator, but it should state the sampling distribution and any importance correction. KL reward shaping changes the advantage before the PPO update, whereas an auxiliary KL loss acts directly on the actor gradient; the two are therefore not equivalent [3].

4.4 Entropy and gradient clipping

Let the vocabulary action set be $\mathcal A$. The policy entropy at valid position $(i,j)$ is

If an entropy bonus is used, define

where $\beta_H\ge0$. The complete actor loss can therefore be written as

Set the coefficient of any disabled regularizer to 0. Let $g$ be the gradient over all parameters updated by the current loss, and let $c_g>0$ be the clipping threshold. Global gradient-norm clipping gives

where $\varepsilon_{\mathrm{norm}}>0$ is a numerical-stability constant. The aggregation rule used for $\mathcal L_{\mathrm{policy}}$, $\mathcal L_{\mathrm{KL}}$, and $\mathcal L_H$ also determines how regularization strength varies with sequence length. The formulas above use token means for KL and entropy. If the policy loss instead uses a sequence mean, either apply the same sequence weighting to the regularizers or state explicitly that their token weighting is intentional.

5. Loss aggregation for long sequences

The PPO token loss is now defined, but an implementation must still decide how to weight samples of different lengths. Let $\ell_{i,j}$ denote any valid-token loss, such as a policy loss or value loss.

5.1 Token mean

Every valid token receives equal weight, so longer sequences contribute more to the total gradient. DAPO’s token-level policy-gradient aggregation follows this path [10].

5.2 Sequence mean

Let $\mathcal B_+:={i:J_i>0}$ be the set of samples containing valid model tokens, and let $B_+:=\lvert \mathcal B_+\rvert>0$. The sequence mean is

It gives each valid sequence approximately equal weight, but each token in a short sequence receives more weight. Either token mean or sequence mean may be reasonable. The denominator must match the intended optimization unit rather than being determined accidentally by microbatch partitioning [3].

In data-parallel training, both numerator and denominator should be reduced globally. If each rank or microbatch first computes a local mean and those means are then averaged while shards contain different numbers of valid tokens, the loss scale changes with the partitioning scheme. When an entire batch contains no valid tokens, the update should be skipped rather than relying on a small epsilon to produce a formally finite but meaningless loss.

6. Asynchronous rollout: four policy roles and correction

6.1 The two ratios in asynchronous PPO

Use four policy roles consistently:

• $\mu$: the behavior policy that actually generated the trajectory;
• $\pi_{\mathrm{old}}$: the proximal policy used for PPO clipping;
• $\pi_\theta$: the current policy being optimized;
• $\pi_{\mathrm{ref}}$: the reference policy used for KL regularization.

Synchronous PPO generally has $\mu=\pi_{\mathrm{old}}$. In asynchronous rollout, a sample may have been generated by an older $\mu$, so it is necessary to distinguish

from

The former constrains the update of the current policy relative to the proximal anchor. The latter reweights each sampled action for the mismatch between the proximal and behavior policies at the recorded prefix. As Section 6.2 explains, a per-token ratio is not a complete correction for the induced state-visitation distribution. The identity

shows why the two can be decoupled. In the bypass configuration described in [13], one sets $\pi_{\mathrm{old}}\equiv\mu$, leaving only $\pi_\theta/\mu$.

6.2 Truncated importance sampling, sequence importance sampling, and rejection

Let $c_{\mathrm{IS}}>0$ be the upper bound for a token importance weight, and let $\ell_{i,j}^{\mathrm{PPO}}$ denote whichever PPO token loss is currently selected, such as $\ell^{\mathrm{clip}}$ or $\ell^{\mathrm{dual}}$. Token-level truncated importance sampling can be written as

Truncation reduces the variance caused by extreme weights but introduces bias. Treat importance weights as fixed rollout data and detach them during optimization. A per-token weight corrects the action distribution conditional on the prefix already visited; it does not correct the distribution of prefixes or visited states. Correcting the distribution through the current position requires accumulating the preceding likelihood ratios.

If tool results, environment randomness, and trajectory-selection rules are held fixed, the joint likelihood ratio for the model-generated action sequence is the product of its token ratios. This ratio does not correct changes in those external mechanisms. Compute the product in log space to avoid numerical underflow:

If the sequence weight is truncated, separately define an upper bound $c_{\mathrm{IS}}^{\mathrm{seq}}>0$ and

The batch estimator must then specify its denominator. For sequence losses $L_i$, ordinary importance weighting uses

whereas self-normalized importance sampling uses

These estimators have different bias and scale. If a normalized or post-rejection denominator is zero, skip the update and report that no samples were retained. The variance of joint weights is generally high for long sequences, so implementations also use clipping, rejection, or mismatch statistics aggregated by sequence [13]. IcePop-like mechanisms retain tokens satisfying

where $0<c_{\min}\le c_{\max}$ [14].

For nonnegative weights ${w_r:r\in\mathcal J}$ over a finite, nonempty index set $\mathcal J$, define the effective sample size (ESS) as

If every weight is 0, define $\operatorname{ESS}=0$. The index set $\mathcal J$ may contain tokens or sequences, but the two granularities must not be mixed in one calculation. Reporting should also state whether the weights are raw, truncated, or post-rejection. Since raw ESS depends on $\lvert \mathcal J \rvert$, also report

with $\operatorname{ESS}_{\mathrm{frac}}:=0$ when all weights are zero. A smaller fractional ESS indicates that fewer samples dominate the weighted estimate.

6.3 Staleness is only a proxy

Let $\nu^{\mathrm{train}}\in\mathbb N$ be the current training-weight version and $\nu_i^{\mathrm{gen}}\in\mathbb N$ the version recorded when generation of sample $i$ began. Assume version numbers are monotonic and $\nu^{\mathrm{train}}\ge\nu_i^{\mathrm{gen}}$. Define

A system may discard samples beyond a threshold or apply importance sampling or rejection to mildly stale data. A fully asynchronous trainer must also pause and resume partial rollouts during parameter synchronization [13][15]. Version distance is only a proxy, however: even at the same version, differences in quantization, kernels, numerical precision, or MoE routing can create log-probability mismatch [13][16].

7. Optional extensions and implementation checks

7.1 Sequence-level policy ratio as a changed policy objective

The $w_i^{\mathrm{seq}}$ above is a joint sequence importance weight for off-policy correction. Group Sequence Policy Optimization (GSPO) instead replaces the token-level ratio with a length-normalized geometric mean of the token ratios inside the policy objective, making each sequence a single optimization unit with sequence-level clipping [17]:

The geometric mean of the token-level ratios is the sequence-likelihood ratio ${\pi_\theta}/{\pi_{\mathrm{old}}}$ over the model-generated tokens, length-normalized by the exponent $1/J_i$, and the clipping in the objective operates on this sequence-level ratio. Geometric-Mean Policy Optimization (GMPO) is not a sequence-ratio method despite the similar name: it keeps the importance ratio at the token level and replaces the arithmetic mean over token objectives with a geometric mean of the clipped per-token objectives, so that no single outlier token can dominate the update [18].

7.2 PPO with an exponentially weighted moving-average anchor (PPO-EWMA)

PPO-EWMA constructs a lagged proximal policy using a normalized exponential moving average [19][20]. Let $n=1,2,\ldots$ index the policy optimizer updates or minibatches at which the EMA model is updated, and let $\beta_{\mathrm{EMA}}\in[0,1)$ be the decay. Initialize $Z_0=1$ and $\bar\theta_0=\theta_0$. After update $n$ produces $\theta_n$, compute

The next update uses

as the proximal anchor. This update order ensures that current parameters $\theta_n$ do not also serve as the anchor in the same loss that produced them. The normalized form provides bias correction in early steps; at steady state, its coefficients approach those of an ordinary EMA. It changes how the proximal anchor is updated, not the reward, GAE timeline, or mask semantics.

7.3 MoE routing replay to align rollout and training

In a mixture-of-experts (MoE) policy, numerical differences may cause the rollout and training routers to select different experts. The resulting policy ratio then mixes parameter changes with routing changes. Let:

• $\ell\in{1,\ldots,L_{\mathrm{MoE}}}$: the MoE layer;
• $r$: an input position in the prefix required to compute a target action;
• $e_{i,r,\ell}^{\mathrm{rollout}}$: the expert ID or top-$k$ expert set selected during rollout at input position $r$ and layer $\ell$ for sample $i$;
• $\mathcal E_{i,\le t}^{\mathrm{rollout}}$: the complete prefix-routing record required to compute action $a_{i,t}$.

Formally, if $\mathcal P(i,t)$ denotes the set of input positions on which the logits for $a_{i,t}$ depend, then

$\mathcal P(i,t)$ includes every prompt, observation, tool, and earlier model-token position needed to compute the logits for $a_{i,t}$; recording only the route at the target position is insufficient. Rollout Routing Replay (R3) fixes these recorded routes during training and computes [16][21]

Routing replay isolates routing mismatch; it cannot repair stale weights, quantization differences, or other kernel differences. If a system replays only the expert IDs or top-$k$ support but recomputes the gate mixture weights, it fixes only expert support and cannot eliminate mixture-weight mismatch.

7.4 Overlong outputs, failed trajectories, and a final checklist

Record separate outcome codes for normal completion, sampling-budget truncation, repetitive loops, tool failures, and environment failures. A soft length penalty may discourage outputs from approaching a hard limit, but a trajectory actually cut off at that limit still requires truncation bootstrapping, explicit filtering, or a separate cutoff marker [10][15].

Before running a multi-turn PPO update, check the following:

1. Action semantics: $m_{i,t}=1$ only for tokens sampled by the model; tool and padding tokens have mask value 0.
2. Reward placement: the terminal reward is placed on the last valid model token, not the end of the tensor.
3. Termination boundary: true termination bootstraps with 0; truncation uses the boundary value.
4. GAE unit: specify whether the unit is a raw token position, compressed model-token decision step, or turn, and whether a mask-0 position advances the trace.
5. Policy roles: verify $\mu=\pi_{\mathrm{old}}$ synchronously; asynchronously, record the behavior/proximal/current/reference policy separately.
6. Loss denominator: specify token mean vs sequence mean and perform a global distributed reduction.
7. Empty samples: samples with $J_i=0$ do not enter objectives that divide by $J_i$; skip the update when the entire batch has no valid tokens.
8. Long-sequence numerics: compute sequence ratios in log space and monitor the weight distribution, ESS, and rejection rate.
9. MoE alignment: if rollout and training routers differ, determine whether the ratio replays identical routes.

References

[1] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017. https://arxiv.org/abs/1707.06347

[2] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv:1506.02438, 2015. https://arxiv.org/abs/1506.02438

[3] verl contributors. PPO Core Algorithms: GAE, Policy Loss, Value Loss, Loss Aggregation and Advantage Whitening. Public source code, commit 0a1c5b64. https://github.com/verl-project/verl/blob/0a1c5b64/verl/trainer/ppo/core_algos.py

[4] verl contributors. Agent Loop: Multi-Turn Tokenization and Response Mask Construction. Public source code, commit 0a1c5b64. https://github.com/verl-project/verl/blob/0a1c5b64/verl/experimental/agent_loop/agent_loop.py

[5] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Second edition, MIT Press, 2018. http://www.incompleteideas.net/book/the-book-2nd.html

[6] Quan Wei et al. Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment. arXiv:2505.11821, 2025. https://arxiv.org/abs/2505.11821

[7] Siliang Zeng et al. Multi-Turn RL Agent Reference Implementation. Public source code. https://github.com/SiliangZeng/Multi-Turn-RL-Agent

[8] Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, and Yang Li. Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs. arXiv:2512.17008, 2025. https://arxiv.org/abs/2512.17008

[9] Yu Yue et al. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv:2504.05118, 2025. https://arxiv.org/abs/2504.05118

[10] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025. https://arxiv.org/abs/2503.14476

[11] Deheng Ye et al. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning. Proceedings of AAAI, 2020. https://arxiv.org/abs/1912.09729

[12] Logan Engstrom et al. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. arXiv:2005.12729, 2020. https://arxiv.org/abs/2005.12729

[13] verl contributors. Mathematical Formulations of Rollout Correction Methods. Public documentation and source-linked implementation. https://verl.readthedocs.io/en/latest/algo/rollout_corr_math.html

[14] InclusionAI et al. Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model. arXiv:2510.18855, 2025. https://arxiv.org/abs/2510.18855

[15] verl contributors. Fully Async Policy Trainer: Staleness Control and Partial Rollout. Public documentation. https://verl.readthedocs.io/en/latest/advance/fully_async.html

[16] Wenhan Ma et al. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers. arXiv:2510.11370, 2025. https://arxiv.org/abs/2510.11370

[17] Qwen Team. Group Sequence Policy Optimization. arXiv:2507.18071, 2025. https://arxiv.org/abs/2507.18071

[18] Yuzhong Zhao et al. Geometric-Mean Policy Optimization. arXiv:2507.20673, 2025. https://arxiv.org/abs/2507.20673

[19] Jacob Hilton, Karl Cobbe, and John Schulman. Batch Size-Invariance for Policy Optimization. arXiv:2110.00641, 2021. https://arxiv.org/abs/2110.00641

[20] OpenAI. PPO-EWMA Reference Implementation. Public source code corresponding to [19]. https://github.com/openai/ppo-ewma

[21] verl contributors. Router Replay Configuration and R3 Integration. Public documentation and source-linked implementation. https://verl.readthedocs.io/en/latest/advance/deepseek_v4_integration.html

[22] Yufeng Yuan et al. What’s Behind PPO’s Collapse in Long-CoT? Value Optimization Holds the Secret. arXiv:2503.01491, 2025. https://arxiv.org/abs/2503.01491

← Back to home