Welcome to verl’s documentation!

verl is a flexible, efficient and production-ready RL training framework designed for large language models (LLMs) post-training. It is an open source implementation of the HybridFlow paper.

verl is flexible and easy to use with:

• Easy extension of diverse RL algorithms: The hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code.

• Seamless integration of existing LLM infra with modular APIs: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM, vLLM and SGLang. Moreover, users can easily extend to other LLM training and inference frameworks.

• Flexible device mapping and parallelism: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.

• Ready integration with popular HuggingFace models

verl is fast with:

• State-of-the-art throughput: By seamlessly integrating existing SOTA LLM training and inference frameworks, verl achieves high generation and training throughput.

• Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

---

Quickstart

• Installation
  • Requirements
  • Choices of Backend Engines
  • Install from docker image
  • Install from custom environment
  • Install with AMD GPUs - ROCM kernel support
• Quickstart: PPO training on GSM8K dataset
  • Introduction
  • Dataset Introduction
  • Step 1: Prepare the dataset
  • Step 2: Download a model for post-training
  • Step 3: Perform PPO training with the instruct model
• Multinode Training
  • Manual
  • Slurm
  • dstack
  • How to debug?
  • Multi-node training on AMD clusters
• Ray Debug Tutorial
  • How to debug?

Programming guide

• HybridFlow Programming Guide
  • Motivation and Design
  • Codebase walkthrough (PPO)
  • Repository organization
• The Design of verl.single_controller
  • Preface
  • Origin
  • A Running Example: generate_sequences
  • Beyond RL Post-Training: Generalizing verl.single_controller

Data Preparation

• Prepare Data for Post-Training
• Implement Reward Function for Dataset

Configurations

• Config Explanation
  • ppo_trainer.yaml for RL FSDP Backend
  • evaluation.yaml
  • sft_trainer.yaml for SFT FSDP Backend

PPO Example

• PPO Example Architecture
• GSM8K Example
• Multi-Modal Example Architecture

Algorithms

• Proximal Policy Optimization (PPO)
• Group Relative Policy Optimization (GRPO)
• Recipe: Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
• Recipe: Self-Play Fine-Tuning (SPIN)
• Recipe: Self-Play Preference Optimization (SPPO)
• Recipe: Entropy Mechanism
• 📃Evaluation
• On-Policy RL with Optimal Reward Baseline (OPO)
• Algorithm Baselines

PPO Trainer and Workers

• PPO Ray Trainer
• PyTorch FSDP Backend
• Megatron-LM Backend
• SGLang Backend

Performance Tuning Guide

• Training DeepSeek 671b
• Performance Tuning Guide
• Upgrading to vLLM >= 0.8
• Resource Needed for verl RL(LoRA)
• NVIDIA Nsight Systems profiling in verl

Adding new models

• Add models with the FSDP backend
• Add models with the Megatron-LM backend

Advanced Features

• Using Checkpoints to Support Fault Tolerance Training
• RoPE Scaling override
• RL(HF) algorithms with LoRA Support
• Multi-turn Rollout Support
• Interaction System for Multi-turn RL Training
• Ray API Design Tutorial
• Extend to other RL(HF) algorithms
• Sandbox Fusion Example

Hardware Support

• Getting started with AMD (ROCM Kernel)
• verl performance tuning for AMD (ROCm Kernel)
• verl x Ascend

API References

• Data interface
• Single Controller interface
• Trainer Interface
• Utilities

FAQ

• Frequently Asked Questions
  • Ray related
  • Distributed training
  • Install related
  • Illegal memory access
  • Checkpoints
  • Triton compile_module_from_src error
  • What is the meaning of train batch size, mini batch size, and micro batch size?
  • How to generate ray timeline to analyse performance of a training job?
  • How to set proxy only for wandb?

Development Notes

• Sandbox Fusion Tool Integration

Contribution

verl is free software; you can redistribute it and/or modify it under the terms of the Apache License 2.0. We welcome contributions. Join us on GitHub, Slack and Wechat for discussions.

Contributions from the community are welcome! Please check out our project roadmap and good first issues to see where you can contribute.

Code Linting and Formatting

We use pre-commit to help improve code quality. To initialize pre-commit, run:

pip install pre-commit
pre-commit install

To resolve CI errors locally, you can also manually run pre-commit by:

pre-commit run

Adding CI tests

If possible, please add CI test(s) for your new feature:

1. Find the most relevant workflow yml file, which usually corresponds to a hydra default config (e.g. ppo_trainer, ppo_megatron_trainer, sft_trainer, etc).

2. Add related path patterns to the paths section if not already included.

3. Minimize the workload of the test script(s) (see existing scripts for examples).

We are HIRING! Send us an email if you are interested in internship/FTE opportunities in MLSys/LLM reasoning/multimodal alignment.