bitsandbytes

bitsandbytes

bitsandbytes is a lightweight wrapper around CUDA custom functions for 8-bit optimizers, matrix multiplication (LLM.int8()), and quantization primitives. It enables fine-tuning and inference of billion-parameter models on a single consumer GPU by reducing memory footprint by 2-4× with minimal performance degradation.

Key Features

• LLM.int8(): 8-bit matrix multiplication that preserves model quality (outlier-aware)
• 4-bit quantization (NF4/FP4): used in QLoRA for memory-efficient fine-tuning
• 8-bit Adam/AdamW: 75% memory reduction for optimizer states
• LoRA + QLoRA integration via PEFT — fine-tune a 65B model on a single 48GB GPU
• Supports PyTorch, Transformers, and Accelerate seamlessly
• Works on CUDA (NVIDIA) and ROCm (AMD) GPUs

Quick Start

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config
)

Install via ai-supply

npx ai-supply add bitsandbytes-quantization

Curated mirror of the open-source bitsandbytes (MIT). Get it from the source.