JARVIS (HuggingGPT) — LLM Task Planner for ML Models
Microsoft Research's system connecting LLMs to 1000+ Hugging Face models for multi-modal task planning and execution.
JARVIS (HuggingGPT)
JARVIS implements the HuggingGPT paper: use an LLM (ChatGPT/GPT-4) as a controller that decomposes multi-modal user requests into sub-tasks, routes each sub-task to the best specialist model on Hugging Face Hub, and synthesises the results into a coherent answer.
Key Features
- Task planning: LLM parses requests into a structured execution DAG
- Model selection: ranks 1000+ HF Hub models by task relevance and download popularity
- Multi-modal: text, image, audio, video tasks in a single pipeline
- Result summarisation: LLM compiles sub-task outputs into a final response
- Gradio UI and REST API included
- Supports local and cloud-hosted specialist models
Quick Start
git clone https://github.com/microsoft/JARVIS
cd JARVIS
pip install -r requirements.txt
# Set OPENAI_API_KEY in .env
python models_server.py --config configs/config.default.yaml &
python app.py --config configs/config.default.yaml
Install via ai-supply
npx ai-supply add jarvis-huggingface-task-planner
Curated mirror of the open-source JARVIS (MIT). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/jarvis-huggingface-task-planner/check). Click a policy:
Consume JARVIS (HuggingGPT) — LLM Task Planner for ML Models programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/jarvis-huggingface-task-planner
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/jarvis-huggingface-task-planner/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add jarvis-huggingface-task-planner
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/jarvis-huggingface-task-planner/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "jarvis-huggingface-task-planner" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.