vLLM V0 to V1: Correctness Before Corrections in RL

TL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 lm_head used for the final projection. We fixed the backend behavior before changing the RL objective.

The reference run used vLLM 0.8.5; the V1 runs used vLLM 0.18.1. vLLM V1 is a substantial rewrite of the V0 engine. Our migration target was therefore deliberately narrow: verify that V1 returned rollout logprobs in the form the trainer expected, rerun the same workload against the V0 reference, and evaluate objective-level changes only after backend parity was restored.

The first visible symptoms appeared in metrics like clamp_log_ratio_new_old_indicator, kl_new_old, entropy, and reward. These came from a GSPO training run. The initial V1 run showed the trainer-side logprobs and reward moving away from the V0 reference early in training.

We separated the possible causes into three layers:

1. Semantic mismatch: the backend returns logprobs with different meaning relative to what the trainer expects.
2. Inference-path mismatch: the backend uses different runtime defaults for caching, scheduling, or request handling.
3. Objective mismatch: the RL objective needs correction for the amount of staleness or backend mismatch.

The first issue was semantic. vLLM V1 returns logprobs from the raw model outputs by default. PipelineRL expected logprobs from the processed distribution used by the sampler. The required setting was logprobs-mode=processed_logprobs.

Next, we addressed runtime defaults. Prefix caching and async scheduling were disabled to match the V0 reference path. In online RL, prefix caching can be problematic if it reuses state computed before a weight update. We also aligned the weight synchronization model using mode="keep" and clear_cache=False during updates.

Finally, numerical parity required matching the precision used for logits. The trainer used an fp32 lm_head for the final projection. Small changes in logits at lower precision become visible in policy ratios and KL. Including the fp32 lm_head path allowed the final V1 run to track the V0 reference perfectly.

The main lesson: fix backend correctness first, then address objective-side improvements like async or off-policy corrections.