LoRA Fine-Tuning Tutorial

The LoRA fine-tuning method in this article comes from the RWKV community fine-tuning project RWKV-PEFT.

Before starting the LoRA fine-tuning, make sure you have a Linux workspace and an NVIDIA graphics card that supports CUDA.

LoRA VRAM Reference

The GPU VRAM requirements for RWKV LoRA fine-tuning can be referred to in the following table:

The data in the above table is based on the following training parameters:

• ctxlen=1024
• micro_bsz=1
• strategy=deepspeed_stage_1
• peft_config='{"r":64,"lora_alpha":32,"lora_dropout":0.05}'

As the training parameters change, the VRAM required for RWKV LoRA fine-tuning will also change.

Collect Training Data

You need to collect binidx data that is more suitable for training RWKV. For specific methods, you can refer to Preparing the Training Dataset.

Configure the Training Environment

To train the RWKV model, you first need to configure the training environment such as conda. For the specific process, please refer to the RWKV Training Environment Configuration section.

Clone the Repository and Install Dependencies

In Linux or WSL, use the git command to clone the RWKV-PEFT repository:

git clone https://github.com/JL-er/RWKV-PEFT.git

After the cloning is completed, use the cd RWKV-PEFT command to enter the RWKV-PEFT directory. And run the following command to install the dependencies required by the project:

pip install -r requirements.txt

Modify the Training Parameters

Open the run_lora.sh file in the RWKV-PEFT/scripts directory using any text editor (such as vscode), and you can modify the training parameters to control the fine-tuning training process and training effect:

The following is a parameter adjustment process for LoRA fine-tuning:

Adjust the Path Parameters

The first three lines of the run_lora.sh file are file path parameters:

• load_model: The path of the base RWKV model
• proj_dir: The output path of the training log and the LoRA file obtained from training
• data_file: The path of the training dataset. Note that there is no need to include the bin and idx suffixes in the path, only the file name is required.

Adjust the n_layer and n_embd Parameters

The following are the corresponding n_layer/n_embd values for RWKV model parameters:

Adjust the Important Training Parameters

Adjust the LoRA-Related Parameters

Adjust Other Training Parameters

The following lists other modifiable training parameters in the script and the effects of their modification.

Appendix: Configuration Reference for run_lora.sh

load_model="/home/rwkv/model/rwkv7-g1-1.5b-20250429-ctx4096.pth"
proj_dir='/home/rwkv/JL/out_model/test'
data_file=/home/rwkv/JL/data/roleplay

n_layer=24
n_embd=2048

micro_bsz=8
epoch_save=1
epoch_steps=200
ctx_len=128
peft_config='{"r":8,"lora_alpha":32,"lora_dropout":0.05}'

python train.py --load_model $load_model \
--proj_dir $proj_dir --data_file $data_file \
--vocab_size 65536 \
--data_type jsonl \
--n_layer $n_layer --n_embd $n_embd \
--ctx_len $ctx_len --micro_bsz $micro_bsz \
--epoch_steps $epoch_steps --epoch_count 4 --epoch_save $epoch_save \
--lr_init 1e-5 --lr_final 1e-5 \
--accelerator gpu --precision bf16 \
--devices 1 --strategy deepspeed_stage_1 --grad_cp 1 \
--my_testing "x070" \
--peft lora --peft_config $peft_config
# Optional parameters
# --op cuda/fla/triton     (Select different operators; if not set, defaults to cuda)
# --wandb RWKV-PEFT-LoRA (Whether to use wandb to monitor the training process)
# --lr_schedule wsd        (Whether to enable cosine annealing for learning rate; 
#                           the default lr_schedule is cos_decay)

Start the Training

In the RWKV-PEFT directory, run the command sh scripts/run_lora.sh to start the LoRA fine-tuning.

After the training starts normally, it should be as follows:

Use the LoRA Weight File

After training is completed, you can find the full LoRA fine-tuned model file (in .pth format) in the output directory.

The merged LoRA fine-tuned model can be used normally in RWKV Runner or Ai00.

For detailed usage, please refer to the Ai00 Tutoria.