SGLang

SGLang

SGLang (Structured Generation Language) is a fast serving framework for large language and vision-language models. Its core innovation is RadixAttention — a prefix tree-based KV cache that enables automatic reuse across requests — dramatically cutting latency for workloads with shared prefixes (system prompts, few-shot examples, RAG contexts).

Key Features

• RadixAttention: automatic KV cache reuse via radix tree, 5× throughput on shared-prefix workloads
• Native structured output: JSON schema, regex, EBNF grammar — zero overhead
• Multi-modal: vision-language models (LLaVA, InternVL, Qwen-VL)
• Speculative decoding and tensor parallelism
• OpenAI-compatible API with streaming
• Supports Llama, Mistral, Qwen, DeepSeek, Gemma, and more

Quick Start

pip install sglang[all]

# Launch server
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000

# Query with structured output
import sglang as sgl

@sgl.function
def classify(s, text):
    s += sgl.user(f"Classify: {text}")
    s += sgl.assistant(sgl.gen("label", choices=["positive", "negative"]))

Install via ai-supply

npx ai-supply add sglang-structured-generation

Curated mirror of the open-source SGLang (Apache-2.0). Get it from the source.