llama.cpp Inference

Thanks to the efforts of RWKV community member @MollySophia, llama.cpp now supports RWKV-6/7 models.

This guide explains how to perform inference with RWKV models using llama.cpp.

Inference with RWKV Models in llama.cpp

Build llama.cpp Locally

You can download precompiled binaries from the llama.cpp releases page.

llama.cpp offers multiple precompiled versions. Choose the appropriate package based on your GPU type:

Alternatively, follow the official llama.cpp build instructions to compile from source.

Obtain GGUF Format Models

llama.cpp uses .gguf format models. Use one of the methods below to obtain a .gguf RWKV model.

python llama.cpp/convert_hf_to_gguf.py ./v6-Finch-1B6-HF  

Run RWKV Model for Chat

Execute the following command in the llama.cpp directory to start a chat session:

./llama-cli -m models/rwkv-6-world-7b-Q8_0.gguf -p "You are a helpful assistant" -cnv -t 8 -ngl 99 -n 500  

This command runs the models/rwkv-6-world-7b-Q8_0.gguf model with 8 threads, initial prompt You are a helpful assistant, and generates up to 500 tokens per response.

Parameter Explanation:

• ./llama-cli: Launches the compiled llama-cli executable.
• -m models/rwkv-6-world-7b-Q8_0.gguf: Path to the model.
• -p "You are a helpful assistant": Initial prompt to start the chat.
• -cnv: Enables chat mode (default, can be omitted).
• -t 8: Thread count (adjust based on available CPU cores).
• -ngl 99: Loads all model layers onto the GPU.
• -n 500: Maximum tokens to generate.

Additional Features (Optional)

Enable completion Mode

./llama-cli -m models/rwkv-6-world-7b-Q8_0.gguf -p "User: What's Spring Festival.\n\nAssistant:" -no-cnv -t 8 -ngl 99 -n 500  

• -p "User: What's Spring Festival.\n\nAssistant:": The "prompt" parameter enables the model to continue writing based on this prompt word.
• -no-cnv: Disables chat mode for completion.

Launch Web Service (Recommended)

Start a web server with:

./llama-server -m models/rwkv-6-world-7b-Q8_0.gguf -ngl 99  

Access the Web UI at http://127.0.0.1:8080:

Quantize GGUF Models

Use ./llama-quantize to quantize fp16 or fp32 GGUF models. Example:

./llama-quantize models/rwkv-6-world-1.6b-F16.gguf models/rwkv-6-world-1b6-Q8_0.gguf Q8_0  

Run ./llama-quantize --help to view all quantization options:

Appendix Code

Convert .pth to HF Format

Use this script to convert .pth models to HF format (generates pytorch_model.bin):

# Convert the model for the pytoch_model.bin  
import torch  

SOURCE_MODEL = "./v6-FinchX-14B-pth/rwkv-14b-final.pth"  
TARGET_MODEL = "./v6-Finch-14B-HF/pytorch_model.bin"  

# Delete target model if exists  
import os  
if os.path.exists(TARGET_MODEL):  
    os.remove(TARGET_MODEL)  

model = torch.load(SOURCE_MODEL, mmap=True, map_location='cpu')  

# Rename keys to match HF format  
new_model = {}  
for key in model.keys():  
    if key.startswith("blocks."):  
        new_key = "rwkv." + key  
        new_key = new_key.replace(".att.", ".attention.")  
        new_key = new_key.replace(".ffn.", ".feed_forward.")  
        new_key = new_key.replace("0.ln0.", "0.pre_ln.")  
    else:  
        new_key = key  
        if key == "emb.weight":  
            new_key = "rwkv.embeddings.weight"  
        if key.startswith("ln_out."):  
            new_key = "rwkv." + key  

    print(f"Renaming key: {key} -> {new_key}")  
    new_model[new_key] = model[key]  

# Save the converted model  
print(f"Saving to: {TARGET_MODEL}")  
torch.save(new_model, TARGET_MODEL)