Dagster
Asset-oriented data orchestration platform for building, testing, and monitoring data pipelines in production.
Dagster
Dagster is an orchestration platform for the development, production, and observation of data assets. It lets you define data pipelines as code with a focus on software-defined assets — the data outputs your pipelines produce.
Key Features
- Software-defined assets: Declare data assets and their dependencies; Dagster determines execution order
- Integrated lineage: Automatically track which assets depend on which, with a visual DAG UI
- Type system: Annotate assets with types and get runtime validation for free
- Partitioned assets: Native support for backfills and incremental processing by partition
- Sensors & schedules: Event-driven triggers and cron-based scheduling
- First-class testing: Unit-test individual ops and assets without running full pipelines
- Ecosystem integrations: dbt, Spark, Snowflake, Databricks, Fivetran, and 100+ others
Quick Start
pip install dagster dagster-webserver
dagster dev
from dagster import asset, Definitions
@asset
def raw_data():
return [1, 2, 3, 4, 5]
@asset
def processed_data(raw_data):
return [x * 2 for x in raw_data]
defs = Definitions(assets=[raw_data, processed_data])
Add to ai-supply
npx ai-supply add dagster-data-orchestrator
Curated mirror of the open-source Dagster (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/dagster-data-orchestrator/check). Click a policy:
Consume Dagster programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/dagster-data-orchestrator
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/dagster-data-orchestrator/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add dagster-data-orchestrator
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/dagster-data-orchestrator/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "dagster-data-orchestrator" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.