JARVIS (HuggingGPT) — LLM Task Planner for ML Models

JARVIS (HuggingGPT)

JARVIS implements the HuggingGPT paper: use an LLM (ChatGPT/GPT-4) as a controller that decomposes multi-modal user requests into sub-tasks, routes each sub-task to the best specialist model on Hugging Face Hub, and synthesises the results into a coherent answer.

Key Features

• Task planning: LLM parses requests into a structured execution DAG
• Model selection: ranks 1000+ HF Hub models by task relevance and download popularity
• Multi-modal: text, image, audio, video tasks in a single pipeline
• Result summarisation: LLM compiles sub-task outputs into a final response
• Gradio UI and REST API included
• Supports local and cloud-hosted specialist models

Quick Start

git clone https://github.com/microsoft/JARVIS
cd JARVIS
pip install -r requirements.txt
# Set OPENAI_API_KEY in .env
python models_server.py --config configs/config.default.yaml &
python app.py --config configs/config.default.yaml

Install via ai-supply

npx ai-supply add jarvis-huggingface-task-planner

Curated mirror of the open-source JARVIS (MIT). Get it from the source.