Crawl4AI
LLM-friendly open-source web crawler that turns pages into clean Markdown/JSON ready for RAG and agent pipelines.
Crawl4AI
Crawl4AI is an open-source, LLM-friendly web crawler and scraper that converts web pages into clean, structured Markdown or JSON ready to feed into RAG and agent pipelines. It is one of the most popular data-ingestion connectors for AI applications, designed for speed and for output that models can consume directly.
Key features
- Fast async crawling built on Playwright with browser session reuse
- LLM-ready Markdown generation with content filtering and pruning
- CSS/XPath selectors plus LLM-based structured extraction strategies
- Handles JavaScript-rendered pages, lazy loading, and stealth/anti-bot options
- Python API and a Docker deployment for scale
Usage note: pip install crawl4ai, run the post-install browser setup, then use the async crawler to fetch a URL and receive cleaned Markdown plus extracted structured data.
Curated mirror of the open-source Crawl4AI (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/crawl4ai-web-crawler/check). Click a policy:
Consume Crawl4AI programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/crawl4ai-web-crawler
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/crawl4ai-web-crawler/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add crawl4ai-web-crawler
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/crawl4ai-web-crawler/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "crawl4ai-web-crawler" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.