TRL (Transformer Reinforcement Learning)

TRL — Transformer Reinforcement Learning

TRL is a full-stack library by Hugging Face for training transformer language models with reinforcement learning from human feedback (RLHF) and related alignment techniques. It provides efficient trainers for every stage of the modern LLM alignment pipeline.

Key Features

• SFTTrainer: supervised fine-tuning with packing, LoRA, and chat templates
• DPO/IPO/KTO: direct preference optimization variants — no reward model needed
• PPO: proximal policy optimization with reward model for classic RLHF
• GRPO: group relative policy optimization (as used in DeepSeek-R1)
• RewardTrainer: train reward models from preference data
• Integrates with PEFT, Accelerate, bitsandbytes for efficient training
• 🤗 Hub model card generation and W&B/TensorBoard logging

Quick Start

from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
trainer = SFTTrainer(
    model=model,
    args=SFTConfig(output_dir="/tmp/sft"),
    train_dataset=dataset,
)
trainer.train()

Install via ai-supply

npx ai-supply add trl-rlhf-training

Curated mirror of the open-source TRL (Apache-2.0). Get it from the source.