Ascend Retool Best Practice

Last updated: 07/03/2026.

引言

Retool论文参考([Retool](https://arxiv.org/pdf/2504.11536)) 集成代码解释器工具，通过多轮实时代码执行进行策略部署，并教会模型根据结果反馈学习何时以及如何调用工具。

1. 环境构建

2. 模型训练

用例模型脚本以及其需要的硬件条件各自如下：

模型	NPU型号	节点数量	训练与推理后端
Qwen2.5-7B	Atlas 900 A2	1	vllm + FSDP

环境构建

1.从自定义Conda环境进行构建

software	version
Python	3.11
CANN	==9.0.0.B160 (CANN900B160)
torch	==2.9.0
torch_npu	==2.9.0
triton_ascend	==3.2.1
verl	main
vllm	v0.18.0
vllm-ascend	v0.18.0
transformers	5.3.0

模型训练与评估

1.模型数据准备

Qwen2.5-7B

下载模型权重

–local-dir: 模型保存路径

git clone https://huggingface.co/Qwen/Qwen2.5-7B-Instruct

下载训练数据集

git clone https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k

下载评估数据集

git clone https://huggingface.co/datasets/Maxwell-Jia/AIME_2024

下载预训练数据集

python3 recipe/retool/retool_sft_preprocess.py

注:自动下载ReTool-SFT，最后生成数据默认保存在~/ReTool-SFT/data目录下

执行预训练脚本

bash recipe/retool/run_qwen2_7b_sft_npu.sh # 需适配脚本中路径

合并预训练权重生成checkpoint

python3 -m verl.model_merger merge --backend fsdp \
    --local_dir /PATH/TO/checkpoint/multiturn-sft-qwen-2.5-7b-instruct/global_step_372 \
    --target_dir /PATH/TO/checkpoint/multiturn-sft-qwen-2.5-7b-instruct/global_step_372/huggingface

2.代码沙箱准备

开源沙箱代码及部署参考 https://github.com/bytedance/SandboxFusion

沙箱代码下载

git clone -b main https://github.com/bytedance/SandboxFusion.git

沙箱安装

cd SandboxFusion
conda create -n sandbox -y python=3.11
conda activate sandbox
pip install poetry
poetry lock
poetry install
mkdir -p docs/build
cd runtime/python
bash install-python-runtime.sh
cd ../../
make run-online

3.训练

示例配置文件如下，在recipe/retool目录下创建一个run_qwen2.5_7b_dapo_npu.sh 根据开发者实际路径配置情况修改模型训练脚本中的以下参数

set -x

export VLLM_USE_V1=1
export TORCHDYNAMO_DISABLE=1
export VLLM_ASCEND_ENABLE_NZ=0
export TASK_QUEUE_ENABLE=1
export VLLM_ENABLE_GRAPH_MODE=1
export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_ASCEND_ENABLE_MLP_OPTIMIZE=1
export LD_PRELOAD=/usr/local/lib/libjemalloc.so.2

# ================= data/model/tool =================
HDFS_ROOT=${HDFS_ROOT:-"${PWD}"}
DATA_ROOT=${DATA_ROOT:-"${PWD}"}

dapo_math_17k=$DATA_ROOT/dataset/BytedTsinghua-SIA/DAPO-Math-17k
aime_2024=$DATA_ROOT/dataset/Maxwell-Jia/AIME_2024
#aime_2025=$DATA_ROOT/dataset/yentinglin/aime_2025
model_path=$DATA_ROOT/dataset/checkpoint/multiturn-sft-qwen-2.5-7b-instruct/global_step_372/huggingface

train_files="['$dapo_math_17k']"
test_files="['$aime_2024']"

# tool
tool_config_path=recipe/retool/sandbox_fusion_tool_config.yaml

# wandb
project_name=retool
experiment_name=qwen2.5-7b_dapo
default_local_dir=$DATA_ROOT/checkpoint/$experiment_name

# 创建日志文件
export TIMESTAMP=$(date +%Y%m%d_%H%M%S)
LOG_DIR="$HDFS_ROOT/verl/logs/$project_name/$experiment_name"
# 判断路径是否存在
if [ ! -d "$LOG_DIR" ]; then
  # 路径不存在，创建路径
  mkdir -p "$LOG_DIR"
  echo "Directory $LOG_DIR created."
else
  echo "Directory $LOG_DIR already exists."
fi

LOG_FILE="${LOG_DIR}/${TIMESTAMP}.log"
touch "$LOG_FILE"
echo "Log file $LOG_FILE created."

# ================= algorithm =================
adv_estimator=grpo

use_kl_in_reward=False
kl_coef=0.0
use_kl_loss=False
kl_loss_coef=0.0

clip_ratio_low=0.2
clip_ratio_high=0.28

max_turns=16
max_prompt_length=2048
max_response_length=20480
actor_lr=1e-6

train_batch_size=32
ppo_mini_batch_size=16

n_resp_per_prompt=16
n_resp_per_prompt_val=30

# ================= performance =================
infer_tp=2 # vllm
train_sp=4 # train
offload=True

actor_max_token_len_per_gpu=$(( (max_prompt_length + max_response_length) * 1 ))
log_prob_max_token_len_per_gpu=$(( actor_max_token_len_per_gpu * 4 ))

PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=$adv_estimator \
  algorithm.use_kl_in_reward=$use_kl_in_reward \
  algorithm.kl_ctrl.kl_coef=$kl_coef \
  data.train_files="$train_files" \
  data.val_files="$test_files" \
  data.return_raw_chat=True \
  data.train_batch_size=$train_batch_size \
  data.max_prompt_length=$max_prompt_length \
  data.max_response_length=$max_response_length \
  data.filter_overlong_prompts=True \
  data.truncation='error' \
  data.custom_cls.path=recipe/retool/retool.py \
  data.custom_cls.name=CustomRLHFDataset \
  custom_reward_function.path=recipe/retool/retool.py \
  custom_reward_function.name=compute_score \
  actor_rollout_ref.model.path=$model_path \
  actor_rollout_ref.model.use_remove_padding=True \
  actor_rollout_ref.model.enable_gradient_checkpointing=True \
  actor_rollout_ref.actor.use_kl_loss=$use_kl_loss \
  actor_rollout_ref.actor.kl_loss_coef=$kl_loss_coef \
  actor_rollout_ref.actor.clip_ratio_low=$clip_ratio_low \
  actor_rollout_ref.actor.clip_ratio_high=$clip_ratio_high \
  actor_rollout_ref.actor.clip_ratio_c=10.0 \
  actor_rollout_ref.actor.optim.lr=$actor_lr \
  actor_rollout_ref.actor.use_dynamic_bsz=True \
  actor_rollout_ref.actor.ppo_mini_batch_size=$ppo_mini_batch_size \
  actor_rollout_ref.actor.ppo_max_token_len_per_gpu=$actor_max_token_len_per_gpu \
  actor_rollout_ref.actor.ulysses_sequence_parallel_size=$train_sp \
  actor_rollout_ref.actor.fsdp_config.param_offload=$offload \
  actor_rollout_ref.actor.fsdp_config.optimizer_offload=$offload \
  actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=$log_prob_max_token_len_per_gpu \
  actor_rollout_ref.rollout.max_num_batched_tokens=$actor_max_token_len_per_gpu \
  actor_rollout_ref.rollout.name=vllm \
  actor_rollout_ref.rollout.mode=async \
  actor_rollout_ref.rollout.max_num_seqs=1024 \
  actor_rollout_ref.rollout.tensor_model_parallel_size=$infer_tp \
  actor_rollout_ref.rollout.multi_turn.enable=True \
  actor_rollout_ref.rollout.multi_turn.max_user_turns=$max_turns \
  actor_rollout_ref.rollout.multi_turn.max_assistant_turns=$max_turns \
  actor_rollout_ref.rollout.multi_turn.tool_config_path=$tool_config_path \
  actor_rollout_ref.rollout.multi_turn.format=hermes \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.9 \
  actor_rollout_ref.rollout.n=$n_resp_per_prompt \
  actor_rollout_ref.rollout.val_kwargs.top_p=0.6 \
  actor_rollout_ref.rollout.val_kwargs.temperature=1.0 \
  actor_rollout_ref.rollout.val_kwargs.n=$n_resp_per_prompt_val \
  actor_rollout_ref.rollout.enable_chunked_prefill=True \
  actor_rollout_ref.rollout.enforce_eager=False \
  trainer.logger=['console'] \
  trainer.project_name=$project_name \
  trainer.experiment_name=$experiment_name \
  trainer.n_gpus_per_node=8 \
  trainer.val_before_train=False \
  trainer.log_val_generations=20 \
  trainer.nnodes=1 \
  trainer.save_freq=100 \
  trainer.default_local_dir=$default_local_dir \
  trainer.test_freq=20 \
  trainer.device=npu \
  actor_rollout_ref.actor.entropy_from_logits_with_chunking=True \
  actor_rollout_ref.ref.entropy_from_logits_with_chunking=True \
  actor_rollout_ref.actor.use_torch_compile=False \
  actor_rollout_ref.ref.use_torch_compile=False \
  actor_rollout_ref.actor.entropy_checkpointing=True \
  actor_rollout_ref.ref.entropy_checkpointing=True \
  actor_rollout_ref.ref.use_torch_compile=False \
  trainer.total_epochs=1 $@ > $LOG_FILE 2>&1 &