Delta Weight Sync

Last updated: 07/10/2026.

Motivation

In a disaggregated setup (hybrid_engine=False) the trainer must broadcast its updated weights to the rollout engine after every step. By default this is a full-weight broadcast whose cost grows with model size. Because RL updates are highly sparse — under typical learning rates over 99% of BF16 weight bytes are unchanged step-over-step — you can instead broadcast only the parameters that changed (a delta), cutting the weight-sync traffic to the sparsity ratio while staying lossless (bit-exact; a per-flush checksum is verified on the receiver).

When to use: disaggregated training with a trainer↔rollout link. Two effects stack here, and they pay off differently:

  • Sparse wire (the “delta” part): only ~1–3% of parameter bytes change per step, so the broadcast payload shrinks accordingly. This effect grows with model size and network distance — on a fast intra-node link with a small model, a full broadcast is already cheap.

  • Shard-local diff + sparse gather (the “sharded” part): no rank ever materializes full tensors or a full-model snapshot, and the gather moves only changed elements. This removes the full-tensor all-gather and rank-0 staging costs that the plain nccl engine pays regardless of network speed — which is why delta_sharded beat the full broadcast at every size we measured (0.5B through 72B, 1.3–3.1×), not just at the large end.

This is why delta_sharded is the only delta backend we ship: an earlier full-gather variant (diff on a rank-0 full-model snapshot) was consistently slower than delta_sharded at every size we measured, so it was dropped in favor of the sharded design.

Design

The delta_sharded backend plugs into the standard checkpoint-engine flow (CheckpointEngineManager → CheckpointEngineWorker), so they work with any trainer that drives weight sync through the checkpoint engine (including the V1 separate_async trainer).

  • Export contract: the trainer’s get_per_tensor_param_shard() yields (name, local_shard, ShardSpec) per local parameter — the spec (see :mod:verl.workers.engine.spec) describes the shard’s placement declaratively (DeviceMesh + Placements), and the engine derives the flat offset, gather group, and contributing rank itself. All layout knowledge stays on the trainer side; the engine is trainer-agnostic.

  • Diff: each rank byte-diffs its own shard against a pinned-CPU snapshot of that shard from the previous sync (no rank holds a full-model snapshot). The comparison is bit-exact (integer view inequality), so the reconstruction is lossless by construction — no thresholds, no drift.

  • Sparse gather + encoding: only the changed (position, value) pairs are gathered to rank 0 (batched, variable-length), translated to full-tensor coordinates, and packed as a shared (positions, values) payload plus a per-parameter manifest (indices encoding: int32 absolute positions).

  • Transport: the sparse payload is broadcast over the existing NCCL collective group in bucket-sized flushes (streamed: each flush is sent and freed as it is produced, so sender peak memory stays ~2 buckets regardless of model size).

  • Apply: each rollout worker hands its local copy of the sparse payload to its colocated SGLang TP worker via same-GPU update_weights_from_tensor IPC, where a verl-shipped loader — registered automatically through SGLang’s stock --custom-weight-loader hook, so no SGLang fork or patch is needed — verifies the flush checksum (fail loud), densifies each parameter’s delta into a NaN-masked tensor, and overwrites only the changed positions in place on the live weights. No full-model mirror is staged anywhere on the rollout side: receiver peak memory is one bucket plus one decode chunk, independent of model size.

  • Seeding: the first sync is an explicit dense pass — the raw weights stream through the same bucketed wire with no positions attached (values only), populating the trainer-side snapshot as they go — so a dummy-initialized rollout gets a correct base without any sparse-encoding overhead. Subsequent syncs are sparse.

Backend

delta_sharded (sharded snapshot)

delta_sharded pushes the diff below the all-gather: each actor rank pins a snapshot of only its FSDP shard, byte-diffs the shard locally, and gathers just the changed (position, value) pairs to rank 0 (via the engine’s get_per_tensor_param_shard() export). So the gather volume drops from the full parameter to the sparsity ratio (~1–3%), and no rank needs a full-model snapshot — the memory and the gather traffic both shard with the world size.

    actor_rollout_ref.rollout.checkpoint_engine.backend=delta_sharded \
    +actor_rollout_ref.rollout.checkpoint_engine.engine_kwargs.delta_sharded.encoding=indices

The assembled delta is bit-identical to full-gather-then-diff, so the wire format, the per-flush checksum, and the rollout-side receiver are all unchanged. Each rank computes its shard’s absolute position in the full flattened parameter purely locally (from the DTensor spec, no extra collective).

Supported training engines: the shard export requires Shard(0) DTensor parameters, which both FSDP versions provide:

  • FSDP2 (fully_shard, actor.strategy=fsdp2): native DTensor params; the export never stages the whole shard on the GPU (state_dict() is reference-only, shards move lazily per parameter).

  • FSDP1 (actor.strategy=fsdp, the default): verl configures SHARDED_STATE_DICT, whose export also emits per-rank Shard(0) DTensors. FSDP1’s state-dict export runs through the unshard machinery, so the whole-shard GPU staging round trip is kept for it (it is skipped for FSDP2). Single-GPU FSDP1 uses FULL_STATE_DICT (plain tensors) and degrades to the replicated/rank-0 path — still correct, just not shard-parallel.

Other shard dimensions than Shard(0) are not supported and raise.

Config note: the training engine reads the top-level actor_rollout_ref.actor.strategy; setting only actor.fsdp_config.strategy does not select FSDP2.

Measured results

All numbers: H100 nodes, GSM8K GRPO, verl V1 separate_async (disaggregated trainer/rollout), FSDP2 + param/optimizer offload, SGLang rollout, per-step steady-state weight sync.

model (placement)

delta_sharded

nccl (full broadcast)

speedup

saved / step

Qwen2.5-7B (1+1 nodes, sustained over 200 steps)

3.8 s

9.1 s

2.4x

5.2 s

Qwen2.5-32B (2+2 nodes)

12.5 s

23.2 s

1.9x

10.7 s

Qwen2.5-72B (4+4 nodes, TP8)

12.0 s

36.9 s

3.1x

24.9 s

The delta sync time stays essentially flat from 32B to 72B – the sharded sparse gather amortizes over the larger trainer world – while the full broadcast grows linearly with parameter bytes, so the advantage widens with scale. The per-step changed ratio is stable at ~1-3% of parameter bytes across sizes and stays there over long runs.

Correctness evidence (details in the PR):

  • 200-step GRPO equivalence at 7B (delta vs nccl, 400 syncs): reward trajectories track phase-for-phase, final rewards within sampling noise, zero receiver checksum failures.

  • Bit-exact round-trip: perturb -> apply as delta -> revert -> apply as delta reproduces greedy generations byte-identically on every prompt.

Usage

A runnable example is verl/experimental/one_step_off_policy/shell/grpo_0.6b_gsm8k_fsdp2_sglang_delta_sharded_2_6.sh — the SGLang 2+6 disaggregated GRPO recipe with backend=delta_sharded.

Current scope: disaggregated (hybrid_engine=False) + SGLang rollout in BF16, FSDP1/FSDP2 training engines. Selecting a delta backend with any other rollout engine raises NotImplementedError at worker startup; a per-backend apply interface (vllm/trt-llm plugins) is planned.

Roadmap

Planned extensions, in design order:

  • Megatron-core trainers: the same delta_sharded backend via a Megatron get_per_tensor_param_shard export whose spec carries the native mcore→HF conversion as a pure-permutation to_hf closure (implemented and validated in a stacked follow-up PR).

  • Quantized rollout (fp8 etc.): diff the quantized bytes (quantize-then-diff) so a low-precision rollout engine can consume deltas without a bf16 intermediate.