RWKV raven avartar

RWKV Language Model

RWKV (pronounced as RwaKuv) is an RNN with GPT-level LLM performance, which can also be directly trained like a GPT transformer (parallelizable).

So it's combining the best of RNN and transformer - great performance, fast inference, fast training, saves VRAM, "infinite" ctxlen, and free sentence embedding. Moreover it's 100% attention-free.

RWKV architecture paper

RWKV paper coveropen in new window

TLDR vs Existing transformer models


  • Lower resource usage (VRAM, CPU, GPU, etc) when running and training.
  • 10x to a 100x lower compute requirements compared to transformers with large context sizes.
  • Scales to any context length linearly (transformers scales quadratically)
  • Perform just as well, in terms of answer quality and capability
  • RWKV models are generally better trained in other languages (e.g. Chinese, Japanese, etc), then most existing OSS models


  • Is sensitive to prompt fomatting, you may need to change how you prompt the model
  • Is weaker at task that require lookback, so reorder your prompt accordingly
    • (e.g. Instead of saying "For the document above do X", which will require a lookback. Say "For the document below do X" instead)

Who sponsors the compute for RWKV?

RWKV is made possible, as an Open Soure project, thanks to the large amount of GPU compute and researchers time contributions from

Without their invaluable support, we would not have been able to develop the core RWKV foundation models that you see today.

In addition, we would like to thank

For helping with GPU time, on smaller experiments, finetunes, and various models. Especially for those models that never get publically released in failed runs.

Quick RWKV community terminology

  • RWKV - The model architecture itself, code found at in new window
  • RWKV World - New base model that is being trained on a larger more diverse mix of dataset, which include samples from over a 100 languages. Partially instruction trained.
  • Raven - Official finetuned version of the base model, with instruction training
  • Base model / Pile Plus Model - RWKV Base model is currently trained on "The Pile" with additional mix of other datasets. This model is not instruction trained.

Which RWKV models should I be using?