Delta Weight Sync

Last updated: 08/23/2026.

Motivation

In a disaggregated setup (hybrid_engine=False) the trainer must broadcast its updated weights to the rollout engine after every step. By default this is a full-weight broadcast whose cost grows with model size. Because RL updates are highly sparse — under typical learning rates over 99% of BF16 weight bytes are unchanged step-over-step — you can instead broadcast only the parameters that changed (a delta), cutting the weight-sync traffic to the sparsity ratio while staying lossless (bit-exact; a per-flush checksum is verified on the receiver).

When to use: disaggregated training with a trainer↔rollout link. Two effects stack here, and they pay off differently:

• Sparse wire (the “delta” part): only ~1–3% of parameter bytes change per step, so the broadcast payload shrinks accordingly. This effect grows with model size and network distance — on a fast intra-node link with a small model, a full broadcast is already cheap.

• Shard-local diff + sparse gather (the “sharded” part): no rank ever materializes full tensors or a full-model snapshot, and the gather moves only changed elements. This removes the full-tensor all-gather and rank-0 staging costs that the plain nccl engine pays regardless of network speed — which is why delta_sharded beat the full broadcast at every size we measured (0.5B through 235B, 1.3–21×), not just at the large end.

This is why delta_sharded is the only delta backend we ship: an earlier full-gather variant (diff on a rank-0 full-model snapshot) was consistently slower than delta_sharded at every size we measured, so it was dropped in favor of the sharded design.

Wire contract: what a rank sends to rank 0

Steady state has ONE canonical shape. Per parameter, each rank contributes a triple (counts[K], idx_concat int32, val_concat) where:

• K is the parameter’s slot count, fixed by a static slot table identical on every rank. Identity params (dense DTensor, explicit blocks, unsharded) have K=1 – the slot is the parameter itself. Slot-enumerable converter params (spec.hf_slots) have one slot per converter output (e.g. a fused gate_up stack: K = E x 2).

• The k-th slice of idx/val holds final HF coordinates inside slot k: the sender has already done all conversion; coordinate semantics never change after this point. Rank 0’s whole job is slot-keyed assembly: concatenate ranks’ (disjoint) pieces per slot, bucket, broadcast. No conversion, no rebuild, no layout knowledge.

Invariants: (1) alignment – every rank enumerates the same K and slot order, with counts[k]=0 for untouched slots, so the batched gather stays in lockstep; (2) disjointness – per slot, different ranks’ coordinate sets never overlap (block geometry guarantees it), so union == concatenation; (3) bounds – a slot has fewer than 2^31 elements (int32 positions), and the batched gather internally splits a round into deterministic sub-rounds (derived from the all-gathered counts matrix, so every rank derives the same split) whenever the largest per-rank blob would exceed bucket_size. Flush triggers are count-only: they must be identical on every rank, and byte totals are not.

Two explicit exemptions: converter params without an enumerable slot table (custom MOE_PARAM_HANDERS, the Megatron shard-list contract) fall back to shard-local payloads plus a rank-0 segmented NaN rebuild; and the dense seed for identity params ships values only (no positions), since the first sync covers every element. The seed for slot params is just the steady shape at full coverage.

Design

The delta_sharded backend plugs into the standard checkpoint-engine flow (CheckpointEngineManager → CheckpointEngineWorker), so they work with any trainer that drives weight sync through the checkpoint engine (including the V1 separate_async trainer).

• Export contract: the delta engine consumes FINAL HF-coordinate payloads; everything backend-specific — the weight→HF naming, the to-HF conversion, the diff and its base — lives on the backend side. The seed (first sync) streams the backend’s existing full export get_per_tensor_param() over the values-only wire: every backend already knows how to assemble and convert its own full tensors (FSDP all-gather, veomni expert restack, Megatron TP/PP fusion), so the seed inherits all of that for free and trainer resume works by construction. After the seed the backend pins its shards (prime_delta_snapshots); every steady sync consumes get_per_tensor_param_delta_shard() — per-parameter entries (slots, dtype_str, counts, hf_idx, hf_val, gather_group) whose coordinates are already final HF coordinates. The engine only batches, gathers, buckets and ships.

• Diff (backend-owned): the default strategy (BaseEngine.get_per_tensor_param_delta_shard via verl.workers.engine.utils.hf_delta_export, so any backend that implements the shard export gets it for free) byte-diffs each rank’s own shard against its pinned-CPU snapshot, refreshed on every export (no rank holds a full-model snapshot). The comparison is bit-exact (integer view inequality), so the reconstruction is lossless by construction — no thresholds, no drift. A backend that already keeps the previous step’s weights (e.g. Decoupled PPO) can diff against that checkpoint instead and skip the dedicated snapshot.

• Sparse gather + encoding: only the changed (position, value) pairs are gathered to rank 0 (batched, variable-length), translated to full-tensor coordinates, and packed as a shared (positions, values) payload plus a per-parameter manifest (indices encoding: int32 absolute positions).

• Transport: the sparse payload is broadcast over the existing NCCL collective group in bucket-sized flushes (streamed: each flush is sent and freed as it is produced, so sender peak memory stays ~2 buckets regardless of model size).

• Apply: each rollout worker forwards its local copy over same-GPU IPC. SGLang uses a VERL loader registered through the stock --custom-weight-loader hook, so no SGLang patch is needed. vLLM uses a VERL WeightTransferEngine adapter and vLLM’s checkpoint patch API. The native model loader still handles checkpoint names, packed parameters, and TP slicing. Both paths verify the checksum and update live weights without keeping a full rollout-model copy.

• Seeding: the first sync is an explicit dense pass — the raw weights stream through the same bucketed wire with no positions attached (values only), populating the trainer-side snapshot as they go — so a dummy-initialized rollout gets a correct base without any sparse-encoding overhead. Subsequent syncs are sparse.

Backend

delta_sharded (sharded snapshot)

delta_sharded pushes the diff below the all-gather: each actor rank pins a snapshot of only its FSDP shard, byte-diffs the shard locally, and gathers just the changed (position, value) pairs to rank 0 (via the engine’s get_per_tensor_param_shard() export). So the gather volume drops from the full parameter to the sparsity ratio (~1–3%), and no rank needs a full-model snapshot — the memory and the gather traffic both shard with the world size.

    actor_rollout_ref.rollout.checkpoint_engine.backend=delta_sharded \
    +actor_rollout_ref.rollout.checkpoint_engine.engine_kwargs.delta_sharded.encoding=indices

The assembled delta is bit-identical to full-gather-then-diff, so the wire format, the per-flush checksum, and the rollout-side receiver are all unchanged. Each rank computes its shard’s absolute position in the full flattened parameter purely locally (from the DTensor spec, no extra collective).

Supported training engines: the shard export requires Shard(0) DTensor parameters, which both FSDP versions and TorchTitan provide:

• FSDP2 (fully_shard, actor.strategy=fsdp2): native DTensor params; the export never stages the whole shard on the GPU (state_dict() is reference-only, shards move lazily per parameter).

• FSDP1 (actor.strategy=fsdp, the default): verl configures SHARDED_STATE_DICT, whose export also emits per-rank Shard(0) DTensors. FSDP1’s state-dict export runs through the unshard machinery, so the whole-shard GPU staging round trip is kept for it (it is skipped for FSDP2). Single-GPU FSDP1 uses FULL_STATE_DICT (plain tensors) and degrades to the replicated/rank-0 path — still correct, just not shard-parallel.

• TorchTitan (model_engine=torchtitan): FSDP2 underneath, with HF names from TorchTitan’s own state dict adapter. HSDP replicate, CP, TP and EP all work — TP and EP cut a dim FSDP2 has already cut, which torch spells _StridedShard but is still one block per rank, and a replicate dim beside them is held fixed rather than spanned. EP differs in naming, not geometry: to_hf() keeps only the locally owned experts, so the export ships the fused stack whole with a slot table naming every one. HSDP with EP also needs a TorchTitan-side fix — its MoE adapter reads placement.dim before checking the placement type, which raises at checkpoint load. PP is rejected at the export boundary: its stages hold disjoint slices, and the gather is lockstep.

Other shard dimensions than Shard(0) are not supported and raise.

Config note: the training engine reads the top-level actor_rollout_ref.actor.strategy; setting only actor.fsdp_config.strategy does not select FSDP2.

Measured results

All numbers: H100 nodes, GSM8K GRPO, verl V1 separate_async (disaggregated trainer/rollout), SGLang rollout, per-step steady-state weight sync. The nccl baseline is current main (including the pinned-staging fix from #7005); param/optimizer offload is ON unless noted.

model (placement)	delta_sharded	nccl (full broadcast)	speedup
Qwen2.5-7B (1+1 nodes)	3.9-4.9 s	5.5-6.0 s	~1.3x
Qwen2.5-32B (2+2 nodes)	11.2-11.9 s	17.7-18.1 s	1.55x
Qwen2.5-32B (2+2 nodes, offload off)	6.2 s	14.2 s	2.3x
Qwen2.5-72B (4+4 nodes, gen TP8, offload off)	12.0-13.0 s	28.5-29.1 s	2.3x
Qwen3-30B-A3B (veomni ep8, 1+1 nodes, 50-step medians)	7.1 s	32.2 s	4.5x
Qwen3-235B-A22B (veomni ep8 x fsdp8, 8+2 nodes, gen TP16)	11.4-14.9 s	246-266 s	~21x

The delta sync time stays essentially flat from 32B through 235B – the sharded sparse gather amortizes over the larger trainer world – while the full broadcast pays a full-model materialization that grows linearly with parameter bytes, so the advantage widens with scale and with MoE sparsity. The per-step changed ratio is ~1-3% of parameter bytes for dense models (0.02-0.05% for the 235B MoE early steps) and stays there over long runs.

The TorchTitan engine and verl’s own FSDP engine (model_engine=dp) were measured separately – A100/A800 80GB, one_step_off_policy, offload off – so they are not comparable to the above:

model (trainer placement)	delta_sharded	nccl (full broadcast)	speedup
Qwen3-8B (2+2 nodes, FSDP2 dp_shard=16)	3.34 s	34.26 s	10.3x
Qwen3-8B (2+2 nodes, FSDP2 dp_shard=8 x TP2)	3.40 s	34.26 s	10.1x
Qwen3-8B (2+2 nodes, HSDP dp_replicate=2 x dp_shard=4 x TP2)	2.41 s	10.24 s	4.2x
Qwen3-8B (1+1 nodes, 50 steps sustained)	3.00 s	11.00 s	3.7x
Qwen3-30B-A3B MoE (2+2 nodes, FSDP2 x EP8, efsdp=2)	8.07 s	43.43 s	5.4x
Qwen3-0.6B (1 node, intra-node NVLink)	0.59 s	0.61 s	1.0x
Qwen3-8B (2+2 nodes, FSDP engine, FSDP2 fsdp_size=16)	2.24 s	38.58 s	17.2x
Qwen3-8B (2+2 nodes, FSDP engine, FSDP1 fsdp_size=16)	24.86 s	65.22 s	2.6x

Neither TP nor HSDP replicas change the payload (0.884%, 0.884% and 0.895% changed across the three 8B TorchTitan rows), and the FSDP engine lands independently on 0.893% for the same config: they change which rank reports an element, not how many moved. Speedups are per session – the HSDP and FSDP rows ran on faster nodes, so compare each only to its own nccl baseline. FSDP1 is the config note above ignored: its export pages the model back to GPU on every sync.

Correctness evidence (details in the PR):

• 200-step GRPO equivalence at 7B (delta vs nccl, 400 syncs): reward trajectories track phase-for-phase, final rewards within sampling noise, zero receiver checksum failures.

• 50-step GRPO equivalence at 30B-A3B (veomni ep8): score trajectories rise in step (0.646->0.719 delta vs 0.639->0.697 nccl), per-step gap at the independent-sampling noise floor, zero checksum failures.

• Bit-exact round-trip: perturb -> apply as delta -> revert -> apply as delta reproduces greedy generations byte-identically on every prompt.

Usage

A runnable example is verl/experimental/one_step_off_policy/shell/grpo_0.6b_gsm8k_fsdp2_sglang_delta_sharded_2_6.sh — the SGLang 2+6 disaggregated GRPO recipe with backend=delta_sharded.

Current scope is disaggregated (hybrid_engine=False) BF16 rollout. The producer side supports FSDP1, FSDP2, and TorchTitan training engines; the rollout consumer may be SGLang or vLLM. The vLLM path requires CUDA IPC, data_parallel_size=1, pipeline_parallel_size=1, a non-quantized model, and vLLM’s checkpoint patch API. For sparse updates, the model loader’s final runtime write must be a floating-point copy_ whose source and destination have the same shape. The vLLM path does not support verify_every > 0, PD disaggregation, speculative decoding, or EPLB. vLLM MoE delta updates require the Triton backend. Selecting any other rollout engine raises NotImplementedError at worker startup.

Roadmap

Planned extensions, in design order:

• Megatron-core trainers: the same delta_sharded backend via a Megatron get_per_tensor_param_shard export. The native mcore→HF converters are whole-param black boxes, outside the dim-0-separable to_hf_chunk contract; the path forward is rewriting them per param family as to_hf_chunk + hf_slots — the main fusions (interleaved qkv, gate_up concat) are row/block permutations and fit the contract, while TP column splits are already expressible as a BlockPlacement dim-1 offset (see #7060).

• Quantized rollout (fp8 etc.): diff the quantized bytes (quantize-then-diff) so a low-precision rollout engine can consume deltas without a bf16 intermediate.