Zero-Logprob RL

November 8, 2026

The RL community is increasingly pursuing zero logprob mismatch between training and generation, even for day-0 models. This article explains why training/generation parity matters for RL and how Megatron Core and Megatron Inference achieve it within NeMo RL.

Why zero logprob mismatch matters

For policy-gradient methods such as REINFORCE 1, PPO 2, and GRPO 3, logprobs connect the reward of a sampled response to an update of the model parameters. A reward can tell us whether an answer is correct or whether generated code passes its tests, but these checks usually provide no differentiable path back through the sampled tokens.

Let cc be a prompt, x=(x1,…,xT)\mathbf{x}=(x_1,\ldots,x_T) a sampled response, and θ\theta the model weights. For a reward R(c,x)R(c,\mathbf{x}) with no direct dependence on θ\theta, the policy-gradient identity gives

∇θEx∼pθ(⋅∣c)[R(c,x)]=Ex∼pθ(⋅∣c)[R(c,x)∇θlog⁡pθ(x∣c)].\nabla_\theta \mathbb{E}_{\mathbf{x}\sim p_\theta(\cdot\mid c)}[R(c,\mathbf{x})] = \mathbb{E}_{\mathbf{x}\sim p_\theta(\cdot\mid c)}\left[ R(c,\mathbf{x})\nabla_\theta\textcolor{blue}{\log p_\theta(\mathbf{x}\mid c)} \right].

For a single on-policy response, the basic sampled policy-gradient loss can now be written using token-level estimates with per-token advantagesa:

L(θ)=−∑t=1TA^t log⁡pθ(xt∣c,x<t)L(\theta)=-\sum_{t=1}^{T}\hat A_t\,\textcolor{blue}{\log p_\theta(x_t\mid c,x_{<t})}

PPO and GRPO losses

PPO and GRPO use probability ratios to optimize a policy using responses sampled by an earlier policy. Let μ\mu be the fixed behavior policy that generated the batch.b The token ratio is:

rt=pθ(xt∣c,x<t)μ(xt∣c,x<t)=exp⁡(log⁡pθ(xt∣c,x<t)−log⁡μ(xt∣c,x<t)).r_t = \frac{p_\theta(x_t\mid c,x_{<t})}{\mu(x_t\mid c,x_{<t})} = \exp\left(\textcolor{blue}{\log p_\theta(x_t\mid c,x_{<t})} - \log\mu(x_t\mid c,x_{<t})\right).

The PPO loss is:

LPPO(θ,ϕ)=E^t[−min⁡(rtA^t,clip⁡(rt,1−ϵclip,1+ϵclip)A^t)].\begin{aligned} L_{\mathrm{PPO}}(\theta,\phi) =\widehat{\mathbb{E}}_t\Big[ &-\min\left(r_t\hat A_t,\operatorname{clip}(r_t,1-\epsilon_{\mathrm{clip}},1+\epsilon_{\mathrm{clip}})\hat A_t\right)\Big]. \end{aligned}

In other words, PPO is trying to maximize the advantage of the sampled tokens, while clipping the advantage to be within a certain range.

For KK responses sampled from μ\mu for one prompt, the original response-normalized GRPO loss uses the group advantages defined above and a direct KL penalty: 3

LGRPO(θ)=1K∑i=1K1Ti∑t=1Ti[−min⁡(ri,tA^i,t,clip⁡(… )A^i,t)+βDKL(pθ(⋅∣si,t)∥pref(⋅∣si,t))].L_{\mathrm{GRPO}}(\theta) =\frac{1}{K}\sum_{i=1}^{K}\frac{1}{T_i}\sum_{t=1}^{T_i} \left[ -\min\left(r_{i,t}\hat A_{i,t},\operatorname{clip}(\dots)\hat A_{i,t}\right) +\beta D_{\mathrm{KL}}\left(p_\theta(\cdot\mid s_{i,t})\|p_{\mathrm{ref}}(\cdot\mid s_{i,t})\right) \right].

Here si,t=(c,x<t(i))s_{i,t}=(c,x_{<t}^{(i)}). The fixed reference policy prefp_{\mathrm{ref}}, typically the initial SFT model, provides the KL baseline; μ\mu is the behavior policy that generated the batch. The coefficient β≥0\beta\geq 0 controls the penalty. For a given prefix ss, the KL divergence is

DKL(pθ(⋅∣s)∥pref(⋅∣s))=Ea∼pθ(⋅∣s)[log⁡pθ(a∣s)−log⁡pref(a∣s)].D_{\mathrm{KL}}\left(p_\theta(\cdot\mid s)\|p_{\mathrm{ref}}(\cdot\mid s)\right) =\mathbb{E}_{a\sim p_\theta(\cdot\mid s)}\left[ \textcolor{blue}{\log p_\theta(a\mid s)}-\log p_{\mathrm{ref}}(a\mid s) \right].

Note that the state ss usually comes from the inference engine, but log⁡pθ(a∣s)\textcolor{blue}{\log p_\theta(a|s)} plus the loss and gradient computation are done in the training loop. If the training framework (e.g., Megatron Core) cannot produce the same logprob values as the generation engine, there will be numeric incorrectness on the model convergence.

In order to achieve this, there are some techniques to address the numeric issues:

TechniqueScopeWhat it establishes
DeterminismTrainingEach repeated runs reproduces itself on the same inputs
Batch invarianceGenerationA request's result does not depend on its neighbors
Training/generation parityBothBoth paths produce the same logprobs at the same weights

In this post, we focus on the third one and discuss how it is implemented in Megatron Core and Megatron Inference with NeMo RL.

NeMo RL recipes

There is a dedicated recipe for zero logprob mismatch in GRPO with Qwen3 in NeMo RL, merged by #3208. It configures Megatron Core, Megatron Inference, and underlying libraries such as Transformer Engine (TE), cuBLAS, and FlashAttention.

Next we're going to why we need these and how they affect the corresponding implementations.

Local arithmetic alignment

First of all, we need to understand the local arithmetic alignment like GEMM, normalization and fused activation.

GEMM and workspace limits

Consider one output element of C=AB\mathbf{C} = \mathbf{A}\mathbf{B}:

cij=∑k=1Kaikbkj.c_{ij} = \sum_{k=1}^{K} a_{ik} b_{kj}.

Changing the batch usually changes MM, while the reduction dimension KK stays fixed. A different MM can nevertheless cause the library to choose a different algorithm, including a different partition of the KK reduction. Floating-point addition does not preserve associativity, so these choices can change the result.

The te_native backend retains native TE/cuBLAS GEMMs and starves their workspace to exclude problematic reduction algorithms, especially split-K variants. It configures the cuBLAS environment and patches TE's own workspace-size functions, clearing their caches.

These controls reach the model's linear projections, including ordinary MLPs, attention projections, and the LM head. The diagram shows attention/Mamba and dense/MoE paths as architecture alternatives; shared experts are optional.

Decoder model with native GEMM projections highlighted in blue and routed-expert grouped GEMMs in orange.

For MoE layers, Megatron Core's optional DeepGEMM path uses BF16 grouped GEMM; te_native keeps native grouped GEMMs. Each output tile processes the full KK dimension in a fixed order, while different output tiles can execute independently.

On Hopper, MMA updates FP32 register accumulators. A full barrier first waits for TMA inputs. After issuing the MMA instructions for a K block, warpgroup_commit_batch() submits the group and warpgroup_wait<0>() waits for all committed groups to finish before the empty barrier releases the input buffer:

On Blackwell, ordered MMA updates accumulate in FP32 TMEM (Tensor Memory). The third argument below selects the same TMEM accumulator throughout the tile's K loop; the fourth enables accumulation after the first MMA:

The epilogue waits for tmem_full before reading the completed tile; tmem_empty prevents premature reuse of its accumulator storage:

Both paths cast to BF16 after the full reduction and stage the result through shared memory before writing it to global memory.

Attention and token alignment

Training reaches FlashAttention through TE, while generation uses Megatron Inference's dynamic attention path. Both select FlashAttention 4 and fix num_splits=1. Split-KV partitions the KV sequence for a query into independently computed segments, then merges their partial outputs using softmax weights. This adds parallelism when a small decode batch has few queries but long KV sequences.

Setting num_splits=1 disables this partitioning and the cross-segment merge, while still processing K/V in tiles, so that the relative order of K/V tiles for one query should be retained as well. It removes one source of batch-dependent numerical differences, but can reduce decode performance when split-KV would improve GPU utilization.

Besides, TE RMSNorm can select a different reduction implementation for a different row-count alignment class. Theoratically, RMSNorm should be batch-invariant since the reduction is performed over the last dimension NN:

RMSNorm⁡(x)ij=xij1N∑j=1Nxij2\operatorname{RMSNorm}(\mathbf x)_{ij} = \frac{\mathbf x_{ij}}{\sqrt{\frac{1}{N}\sum_{j=1}^{N}\mathbf x_{ij}^2}}

Neverthless, the implementation in practice is variant like following:

There are 2 RMSNorm versions in TE 2.18 rmsnorm_fwd_general_kernel and rmsnorm_fwd_tuned_kernel. The former firstly accumulates x2x^2 and then performs the reduction, while the latter accounts the average and variance first. Since E[x2]=E[x]2+Var⁡[x]\mathbb{E}[x^2] = \mathbb{E}[x]^2 + \operatorname{Var}[x], they are mathematically equivalent but the float accumulation order is different, leading to different numerical results.

To avoid this, Megatron Core rounds the number of tokens (MM dimension) into multiple of 64, and applies this strategy to both eager generation and CUDA-graph token buckets.

The final log-softmax

Even identical logits can produce different logprobs if their normalization uses a different addition tree. Training commonly uses vocab-parallel log-softmax: each TP rank computes local normalization terms, then combines them across ranks. Generation commonly normalizes a complete vocabulary row.

In current NeMo RL implementation, _GenAlignedLogprob / _FullVocabLogprob replace the forward of DistributedLogprob and ChunkedDistributedLogprob:

TP vocabulary shards are all-gathered in vocabulary order, converted to FP32, and normalized using the same full-row log-softmax as generation before selecting the target logprob.
NeMo RL's full-vocabulary forward alignment for one sequence position with TP=2. Each small box is a vocabulary logit; the highlighted entry is the target token. All-gather concatenates logits, while the shared log-softmax implementation fixes their reduction order.

The all-gather concatenates values without summing them. Both sides can then use the same full-row reduction, implemented as one program per row with a fixed 1024-column scanning block. Changing the number of rows does not change how a particular row's columns are grouped.

The cost is additional vocabulary communication, temporary memory, and backward recomputation. Sequence chunking limits peak memory in the chunked path. The strict recipe rejects use_fused_linear_logprobs because it bypasses this shared forward.

Megatron Core PR #8421 proposes a separate approach for native vocab-parallel cross entropy: fixed local sums followed by an ordered sum across vocabulary shards. It does not replace NeMo RL's logprob functions or the engine's log_softmax. The vocab-parallel cross entropy discussion below explains why these paths still need to be matched explicitly.

Distributed reductions

All-gather, all-to-all, and pipeline send/recv primarily move or reorder values. With a unique source for each element and correct buffer synchronization, packet arrival order does not define a floating-point sum. All-reduce, reduce-scatter, scatter-add, and MoE combine do perform sums, so their grouping, precision, and rounding points need to agree.

TP at a fixed size

Let PP denote the tensor-parallel size. Current NeMo RL baseline requires the same PP for training and generation, with a limit of P≤2P\leq 2. At TP=2, generation disables the NVLS TP reduction fast path and uses NCCL to match training.

TP reduction mechanism

A row-parallel layer produces one partial per rank for each output element. NCCL can change their addition order with message size, chunking, or the element's position in the message. The same token can therefore follow different addition trees in a training batch and a decode batch.

Megatron Core PR #8421 (unmerged; reviewed at 72c11fb) proposes an ordered mode that fixes the order. For partial uru_r from rank rr, it computes

a0=FP32(u0),ar+1=roundFP32(ar+FP32(ur+1)),y=castD(aP−1),\begin{aligned} a_0 &= \mathrm{FP32}(u_0), \\ a_{r+1} &= \mathrm{round}_{\mathrm{FP32}}\left(a_r+\mathrm{FP32}(u_{r+1})\right), \\ y &= \mathrm{cast}_{D}(a_{P-1}), \end{aligned}

where DD is the output dtype. Every FP32 addition still rounds, but the sequence of additions is fixed and there is only one final cast to DD. The result depends on the partials, rather than where the token sits in the batch.

Training and generation use different data-transfer strategiesc to execute this same sum:

TP=4 ordered reduce-scatter: training pushes partials into the output owner's staging slots, while generation pulls them from peer buffers. Both fold source ranks 0, 1, 2, 3 in FP32 before the final cast.

Matching different TP sizes

Fixed rank order alone does not make TP=2 and TP=4 identical. Changing PP changes each rank's GEMM reduction length from K/PK/P to a different value, so the partials themselves can change before communication begins.

The opt-in batch_invariant_tp_shards=G instead divides the row-parallel reduction dimension KK into GG fixed logical shards. Each shard produces its own GEMM partial qjq_j, independent of which rank owns it. For example, with G=4G=4:

Physical TP sizeShards assigned to each rank
P=4P=4Rank 0: q0q_0; rank 1: q1q_1; rank 2: q2q_2; rank 3: q3q_3
P=2P=2Rank 0: q0,q1q_0,q_1; rank 1: q2,q3q_2,q_3
P=1P=1Rank 0: q0,q1,q2,q3q_0,q_1,q_2,q_3

All three layouts evaluate the same ordered fold, ((q0+q1)+q2)+q3((q_0+q_1)+q_2)+q_3, with the same accumulator precision and final cast. When P<GP<G, each rank computes its G/PG/P partials with one strided batched GEMM. It must retain and communicate the individual partials: pre-summing them within each rank would change the global addition tree. At P=1P=1, the fold is entirely local; at P=GP=G, each rank owns one shard and uses the corresponding single-partial path.

This requires the same GG on both sides, GG divisible by each physical TP size, and KK divisible by GG. The multi-shard path adds partial-buffer and communication costs, even though the GEMMs remain cuBLAS operations.

Vocab-parallel cross entropy

The PR also makes Megatron Core's native vocab-parallel cross entropy use a fixed reduction for its softmax denominator. For a vocabulary logit zvz_v, let m=max⁡vzvm=\max_v z_v and let Vj\mathcal{V}_j be vocabulary shard jj. The denominator is computed as

dj=FixedSumv∈Vjexp⁡(zv−m),Z=OrderedFold(d0,…,dG−1).d_j = \mathrm{FixedSum}_{v\in\mathcal{V}_j}\exp(z_v-m), \qquad Z = \mathrm{OrderedFold}(d_0,\ldots,d_{G-1}).

Without an explicit GG, there is one vocabulary shard per TP rank, so G=PG=P for this calculation. With batch_invariant_tp_shards=G, the same GG vocabulary shards are used at every supported TP size. Both sides must use the same padded vocabulary size, divisible by GG.

Within each shard, a Triton program uses fixed 1024-lane accumulation followed by a fixed tree sum. Across shards, it uses the ordered fold described above. For ordinary finite logits, the maximum reduction and target-logit selection do not have the same summation-order problem: max is order-independent, and only the target's owning rank contributes a nonzero selected logit.

EP: reproducing the same two-level sum

Expert parallelism (EP) places different experts on different GPUs. For each token, dispatch sends its inputs to the selected experts, and combine adds their contributions back into one output. Compared with the TP reduction above, EP has another level of summation: several selected experts can live on the same rank.

An atomic scatter-add can make the result depend on which contribution arrives first. Even two deterministic implementations can disagree if they use different addition trees. Training and generation must therefore agree on both the local expert sum and the cross-rank sum, including their intermediate casts.

For the common path where routing probabilities have already been applied inside the activation, let yt,e\mathbf{y}_{t,e} be expert ee's contribution to token tt. There is no additional probability multiplication during combine. First, each rank adds its selected experts in fixed expert order using an FP64 accumulator, then casts the partial to FP32. For the ordered set of selected experts Er(t)E_r(t) on rank rr:

ur(t)=FP32(∑e∈Er(t)expert orderFP64(yt,e)).(2)\mathbf{u}_r(t) = \mathrm{FP32}\left( \sum_{e \in E_r(t)}^{\mathrm{expert\ order}} \mathrm{FP64}(\mathbf{y}_{t,e}) \right). \tag{2}

Second, an FP32 accumulator adds these partials in ascending rank order, then the result is cast to BF16. With PEPP_{\mathrm{EP}} EP ranks:

a0(t)=0,ar+1(t)=FP32(ar(t)+ur(t)),v(t)=BF16(aPEP(t)).(3)\begin{aligned} \mathbf{a}_0(t) &= \mathbf{0}, \\ \mathbf{a}_{r+1}(t) &= \mathrm{FP32}\left(\mathbf{a}_r(t) + \mathbf{u}_r(t)\right), \\ \mathbf{v}(t) &= \mathrm{BF16}\left(\mathbf{a}_{P_{\mathrm{EP}}}(t)\right). \end{aligned} \tag{3}

Training dispatches tokens and returns expert outputs through all-to-alld. Its batch-invariant unpermute uses an inverse map from each token to its expert-output rows, so each output reads and adds its own contributions without competing atomic writes. It groups those rows by EP rank and reproduces (2) and (3) inside the combine computation.

Generation with EP greater than 1 uses the NVLS dispatcher:

  1. AllGather-V collects variable-count tokens and routing metadata.
  2. Each rank runs its local experts and writes their combined FP32 partials to a symmetric-memory buffer.
  3. After synchronization makes peer data readable, ordered_reduce_scatter_v reads the partials and adds source ranks in ascending order using FP32.
  4. Each token's owner receives its combined rows, then casts them to BF16.

EP=2 generation for one token owned by rank 0: AllGather-V shares its input, each rank forms a local FP32 expert partial, and rank 0 combines the ready partials in rank order before casting to BF16.
One token selects experts A and B on rank 0, and C and D on rank 1. Routing probabilities are already applied upstream; the token's owner reads both rank partials in a fixed order.

Perf tax and results

For Qwen3-30B-A3B at EP=8 and TP=1, Megatron Core PR #6521 and NeMo RL PR #3531 report roughly 1-2% generation overhead with batch invariance and training/generation parity. These generation measurements do not establish the total GRPO overhead, which depends on the model, parallel layout, and kernel choices. Validation combines exact token-level logprob comparisons with multi-step synchronous GRPO, requiring gen_kl_error == 0 and token_mult_prob_error == 1 across weight updates. NeMo RL PR #4683 reports about 248,000 response tokens with identical training/generation logprobs over five steps for a Nemotron Nano v3 30B-A3B MXFP8 configuration with sequence packing.

Footnotes

  1. Ronald J. Williams (1992), Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning 8, 229-256. ↩

  2. John Schulman et al. (2017), Proximal Policy Optimization Algorithms, arXiv:1707.06347. ↩

  3. Zhihong Shao et al. (2024), DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, arXiv:2402.03300; introduces Group Relative Policy Optimization. ↩ ↩2