Trainer Interface

Last updated: 06/08/2025 (API docstrings are auto-generated).

Trainers drive the training loop. Introducing new trainer classes in case of new training paradiam is encouraged.

Core APIs

verl.utils.tokenizer.hf_tokenizer(name_or_path, correct_pad_token=True, correct_gemma2=True, **kwargs)[source]

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens.

Parameters:

  • name (str) – The name of the tokenizer.

  • correct_pad_token (bool) – Whether to correct the pad token id.

  • correct_gemma2 (bool) – Whether to correct the gemma2 tokenizer.

Returns:

The pretrained tokenizer.

Return type:

transformers.PreTrainedTokenizer

Core functions to implement PPO algorithms. The function implemented in this file should be used by trainer with different distributed strategies to implement PPO-like algorithms.

verl.trainer.ppo.core_algos.agg_loss(loss_mat: Tensor, loss_mask: Tensor, loss_agg_mode: str, dp_size: int = 1, batch_num_tokens: int | None = None, global_batch_size: int | None = None, loss_scale_factor: int | None = None)[source]

Aggregate the loss across global batch to ensure the loss is invariant to fsdp/megatron parallelism.

NOTE: The returned loss has different behaviors for different backend: - FSDP: the loss is directly used for backward. - Megatron: the loss should be scaled by num_microbatches and cp_size for pp schedule.

Parameters:

  • loss_mat – micro batch loss matrix, (bs, response_length)

  • loss_mask – micro batch loss mask, (bs, response_length)

  • loss_agg_mode – method to aggregate the loss matrix into a scalar

  • dp_size – data parallel size

  • batch_num_tokens – number of valid tokens in global batch

  • global_batch_size – global batch size

  • loss_scale_factor – scale factor for “seq-mean-token-sum-norm” mode. If None, uses loss_mask.shape[-1]. Set this to a constant value to ensure consistent normalization throughout training.

Returns:

a scalar torch.Tensor

aggregated loss

Return type:

loss

verl.trainer.ppo.core_algos.compute_policy_loss(old_log_prob, log_prob, advantages, response_mask, cliprange=None, cliprange_low=None, cliprange_high=None, clip_ratio_c=3.0, loss_agg_mode: str = 'token-mean')[source]

Compute the clipped policy objective and related metrics for PPO.

Adapted from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1122

Parameters:

  • old_log_prob (torch.Tensor) – Log-probabilities of actions under the old policy, shape (batch_size, response_length).

  • log_prob (torch.Tensor) – Log-probabilities of actions under the current policy, shape (batch_size, response_length).

  • advantages (torch.Tensor) – Advantage estimates for each action, shape (batch_size, response_length).

  • response_mask (torch.Tensor) – Mask indicating which tokens to include in the loss, shape (batch_size, response_length).

  • cliprange (float, optional) – Clipping parameter ε for standard PPO. See https://arxiv.org/abs/1707.06347. Defaults to None (must be provided).

  • cliprange_low (float, optional) – Lower clip range for dual-clip PPO. Defaults to same as cliprange.

  • cliprange_high (float, optional) – Upper clip range for dual-clip PPO. Defaults to same as cliprange.

  • clip_ratio_c (float, optional) – Lower bound of the ratio for dual-clip PPO. See https://arxiv.org/pdf/1912.09729. Defaults to 3.0.

  • loss_agg_mode (str, optional) – Aggregation mode for agg_loss. Defaults to “token-mean”.

verl.trainer.ppo.core_algos.kl_penalty(logprob: FloatTensor, ref_logprob: FloatTensor, kl_penalty) → FloatTensor[source]

Compute KL divergence given logprob and ref_logprob. Optionally using straight through to bind k2 on other kl penalty compute method for unbiased KL gradient estimation. See more description in http://joschu.net/blog/kl-approx.html

Parameters:

  • logprob

  • ref_logprob

Returns:

kl_estimate

verl.trainer.ppo.reward.load_reward_manager(config: DictConfig, tokenizer: Any, **reward_kwargs: Any) → RewardManagerBase[source]

Load and initialize a reward manager based on the configuration.

Parameters:

  • config – PPO trainer configuration object containing reward_model fields.

  • tokenizer – Tokenizer object used for processing text.

  • **reward_kwargs – Additional keyword arguments for the reward manager.

Returns:

An instance of the specified reward manager class.

class verl.workers.reward_manager.NaiveRewardManager(tokenizer, num_examine, compute_score=None, reward_fn_key='data_source', compute_score_timeout=None)[source]

The reward manager.

class verl.workers.reward_manager.DAPORewardManager(tokenizer, num_examine, compute_score=None, reward_fn_key='data_source', max_resp_len=None, overlong_buffer_cfg=None)[source]

The reward manager.