scanned 1mo ago·osv · gitleaks · opengrep · picklescan + heuristics·full breakdown in the Security tab ↓
GPT Crawler
GPT Crawler (by Builder.io) crawls a website starting from one or more seed URLs and packages the extracted content into a single knowledge file you can upload to a custom GPT, assistant, or RAG index. You give it a URL match pattern and a CSS selector for the main content, and it walks the site with a headless browser — respecting page limits and concurrency — then emits an output.json ready for ingestion.
Key features
Point at a docs site or blog and get one clean knowledge file
Configurable URL match globs, content selector, max pages, and depth
Headless-browser rendering for JavaScript-heavy pages
Run as a CLI, a dependency, or inside a container
Output tailored for OpenAI custom GPTs and assistant uploads
A focused, practical tool for turning existing documentation into an LLM-ready corpus without building a full crawling stack from scratch.
Curated mirror of the open-source GPT Crawler (ISC). Get it from the source.
✓ Security: Safe · 100100/100 · grade Ascanned 1mo ago
✓ no compromise signals5 risk-surface · 3/20 OWASP controls flagged
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
What this capability can do · med confidence (static)
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
OWASP Top 10 for LLM Applications
⚠LLM03Supply Chainhigh
Vulnerable/compromised dependencies, models or archives in the artifact.