Welcome to verl’s documentation!

verl is a flexible, efficient and production-ready RL training framework designed for large language models (LLMs) post-training. It is an open source implementation of the HybridFlow paper.

verl is flexible and easy to use with:

• Easy extension of diverse RL algorithms: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code.

• Seamless integration of existing LLM infra with modular APIs: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks.

• Flexible device mapping and parallelism: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.

• Ready integration with popular HuggingFace models

verl is fast with:

• State-of-the-art throughput: By seamlessly integrating existing SOTA LLM training and inference frameworks, verl achieves high generation and training throughput.

• Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

---

Quickstart

• Installation
  • Requirements
  • Choices of Backend Engines
  • Install from docker image
  • Install verl-SGLang from scratch
  • Install from custom environment
  • Install with AMD GPUs - ROCM kernel support
    • Find the docker for AMD ROCm: docker/Dockerfile.rocm
    • Build the image:
    • Launch the container
• Quickstart: PPO training on GSM8K dataset
  • Introduction
  • Dataset Introduction
  • Step 1: Prepare the dataset
  • Step 2: Download a model for post-training
  • Step 3: Perform PPO training with the instruct model
• Multinode Training
  • Manual
    • Set up multinode ray cluster
    • Submit job to ray cluster
  • Slurm
  • How to debug?
    • Legacy Ray Debugger
    • Ray Distributed Debugger VSCode Extension
  • Multi-node training on AMD clusters
    • slurm_script.sh
    • Run multi-node training with above slurm_script.sh

Programming guide

• HybridFlow Programming Guide
  • Motivation and Design
    • DataFlow
      • RL as a dataflow problem
    • Design Choices
    • Overall Execution Diagram
  • Codebase walkthrough (PPO)
    • Entry function
    • Ray trainer
    • Worker and WorkerGroup construction
    • Worker definition
    • PPO main loop
    • Takeaways
  • Repository organization

Data Preparation

• Prepare Data for Post-Training
• Implement Reward Function for Dataset
  • RewardManager
  • Reward Functions
    • Pre-implemented
    • Customized

Configurations

• Config Explanation
  • ppo_trainer.yaml for FSDP Backend
    • Data
    • Actor/Rollout/Reference Policy
    • Critic Model
    • Reward Model
    • Customized Reward Function
    • Algorithm
    • Trainer

PPO Example

• PPO Example Architecture
  • Define the data
  • Define the reward functions for different datasets
  • Define worker classes
  • Defining reward model/function
  • Define, init and run the PPO Trainer
• GSM8K Example
  • Introduction
  • Dataset Introduction
  • Step 1: Prepare dataset
  • Step 2: Download Model
  • [Optional] Step 3: SFT your Model
  • Step 4: Perform PPO training with your model on GSM8K Dataset

PPO Trainer and Workers

• PPO Ray Trainer
• PyTorch FSDP Backend
• Megatron-LM Backend

Performance Tuning Guide

• Performance Tuning Guide
• Upgrading to vLLM >= 0.8

Experimental Results

• Algorithm Baselines

Advance Usage and Extension

• Ray API Design Tutorial
• Extend to other RL(HF) algorithms
• Add models with the FSDP backend
• Add models with the Megatron-LM backend
• Using Checkpoints to Support Fault Tolerance Training

API References

• Data interface

FAQ

• Frequently Asked Questions

Contribution

verl is free software; you can redistribute it and/or modify it under the terms of the Apache License 2.0. We welcome contributions. Join us on GitHub, Slack and Wechat for discussions.

Code formatting

We use yapf (Google style) to enforce strict code formatting when reviewing MRs. Run yapf at the top level of verl repo:

pip3 install yapf
yapf -ir -vv --style ./.style.yapf verl examples tests