A system to programmatically author, schedule, and monitor data pipelines.
PythonApache License 2.0active
DataEngineering Workflow
Search all DataEngineering projects →13 projectsWorkflow
- 92%active
- 8%dormant
Workflow
Scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
JavaApache License 2.0active
A Python module that helps you build complex pipelines of batch jobs.
PythonApache License 2.0active
An open-source Python library for building data applications.
PythonApache License 2.0active
Open-source platform for data preparation, synthetic data generation, and AI/data pipelines. Includes reusable skills for automating workflow steps across data and AI tasks.
PythonApache License 2.0active
A warehouse-first Customer Data Platform that enables you to collect data from every application, website and SaaS platform, and then activate it in your warehouse and business tools.
GoOtheractive
The open-source reverse ETL, data activation platform for modern data teams.
RubyGNU Affero General Public License v3.0active
End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.
GoApache License 2.0active
DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs.
JavaScriptApache License 2.0dormantarchived
Intent-as-code workflow engine for AI data pipelines: reviewable YAML DAGs statically checked (schema, permits, cost floor) before execution, with tamper-evident run traces.
RustGNU Affero General Public License v3.0active
An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed, regardless of the data platform (Snowflake, BigQuery, DataBricks, etc.)
KotlinApache License 2.0active
Self-hosted gateway for safe, auditable queries for agents across approved data sources.
TypeScriptApache License 2.0active
A lightweight Python library for building execution pipelines with retry, parallel execution, cron scheduling, and async support.
PythonApache License 2.0active
Back to Awesome DataEngineeringSearch Awesome DataEngineering →