dlt (data load tool)
Open-source Python library for building self-maintaining, schema-evolving data pipelines with minimal code.
dlt — data load tool
dlt is an open-source Python library that makes loading data from any source into any destination simple and production-ready. It handles schema inference, evolution, nested data normalization, incremental loading, and secrets management automatically.
Key Features
- Zero-schema setup: dlt infers schema from your data automatically and evolves it as your source changes
- Nested normalization: JSON arrays and objects are flattened into normalized relational tables
- Incremental loading: Built-in cursor-based and append/merge strategies with state management
- 100+ verified sources: REST APIs, databases, SaaS tools, files, and cloud storage via the dlt Hub
- Destination-agnostic: DuckDB, BigQuery, Snowflake, Redshift, Postgres, Delta Lake, and more
- Secrets management: Native integration with .env, Vault, AWS Secrets Manager, and GCP Secret Manager
Quick Start
pip install dlt[duckdb]
import dlt
@dlt.resource
def github_events():
import requests
yield from requests.get(
"https://api.github.com/events"
).json()
pipeline = dlt.pipeline(
pipeline_name="github",
destination="duckdb",
dataset_name="events"
)
pipeline.run(github_events())
Add to ai-supply
npx ai-supply add dlt-data-load-tool
Curated mirror of the open-source dlt (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/dlt-data-load-tool/check). Click a policy:
Consume dlt (data load tool) programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/dlt-data-load-tool
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/dlt-data-load-tool/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add dlt-data-load-tool
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/dlt-data-load-tool/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "dlt-data-load-tool" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.