A key-value store for microcontroller and IoT applications.
CBSD 3-Clause "New" or "Revised" Licenseslowing
Awesome DataEngineering
Search 112 DataEngineering projects →112 projects16 categoriesshowing 61–112, page 2 of 2
- 55%active
- 14%steady
- 10%slowing
- 21%dormant
Sourced fromigorbarinov/awesome-data-engineering
Projects 61–112 of 112, most starred first
Distributed Transactional In-Memory Database (based on MongoDB).
JavaScriptApache License 2.0dormant
Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.
Scalasteady
Data Validation Tool compares data from source and target tables to ensure that they match. It provides column validation, row validation, schema validation, custom query validation, and ad hoc SQL exploration.
PythonApache License 2.0active
Simplified command-line administration for Kafka brokers.
RubyApache License 2.0dormant
In-process analytical SQL database written in C++20. Reads Parquet, CSV, JSON, Avro, Arrow, SQLite, and Excel directly. Single binary, Python package, and 1.3 MB WASM build for the browser.
C++MIT Licenseactive
Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated.
GoOtherdormantarchived
A cloud native data pipeline and transformation toolkit written in Go.
GoMIT Licensesteady
Open-source SQL to map platform for BigQuery, Snowflake, and PostGIS.
JavaScriptGNU Affero General Public License v3.0active
A time series database application that provides secure access to time series data based on Accumulo and Grafana.
JavaApache License 2.0steady
A free & cross platform monitoring tool (Spark UI / Spark History Server alternative).
ScalaOtherslowingarchived
A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production.
PythonMIT No Attributionsteady
Scalable machine learning library for Hive/Hadoop.
JavaApache License 2.0dormantarchived
A Kafka Proxy, solving problems like encrypting your Kafka data at rest.
JavaApache License 2.0active
A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2).
ShellMIT Licensedormant
A database for user interactions (likes, views, follows) represented as graphs, with precomputed reads served in real-time.
KotlinApache License 2.0active
A lightweight tool for easy deployment and rollback of dockerized applications.
GoApache License 2.0dormant
Connecting Apache Spark with different data stores. Deprecated.
JavaApache License 2.0dormantarchived
An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying full automation of data quality monitoring.
JavaOthersteady
The world's most comprehensive authoritative data source knowledge base. 160+ curated sources from governments, international organizations, and research institutions with MCP integration.
PythonMIT Licenseactive
A PostgreSQL extension to produce messages to Apache Kafka.
CMIT Licensedormantarchived
Python PDF-to-Markdown orchestrator. Classifies each page and routes to the optimal backend (PyMuPDF, Docling, RapidOCR, Gemini Flash), emitting Markdown plus a per-page confidence score so ingestion pipelines can quarantine low-trust pages before feeding LLMs or retrieval.
PythonMIT Licenseactive
Decorator-first DataFrame contracts/validation (columns/dtypes/constraints) at function boundaries. Supports Pandas/Polars/PyArrow/Modin.
PythonMIT Licenseactive
Personal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks, ancestry, pharmacogenomics, nutrition, psychology, etc.) and generating a terminal-style single-page HTML visualization.
PythonMIT Licensesteady
Intent-as-code workflow engine for AI data pipelines: reviewable YAML DAGs statically checked (schema, permits, cost floor) before execution, with tamper-evident run traces.
RustGNU Affero General Public License v3.0active
Python SDK that dispatches parallel web-research agents across table rows, synthesizing multi-agent findings into structured columns.
PythonMIT Licenseactive
Kafka-winston logger for Node.js from Uber.
JavaScriptMIT Licensedormantarchived
A Snowflake-compatible emulator for local development and testing.
GoMIT Licenseactive
An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed, regardless of the data platform (Snowflake, BigQuery, DataBricks, etc.)
KotlinApache License 2.0active
Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, Seaweed-FS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".
GoApache License 2.0active
OSS Reverse ETL CLI. Sync data from warehouses to business tools via YAML.
PythonApache License 2.0active
Live import all your Google Sheets to your data warehouse.
PHPMIT Licensesteady
A vendor-neutral, declarative data quality engine. Define checks in YAML, run anywhere. Includes 16 built-in check types, SQL batch optimizer, anomaly detection, and data contracts.
PythonApache License 2.0active
Self-hosted gateway for safe, auditable queries for agents across approved data sources.
TypeScriptApache License 2.0active
High-performance, streaming-first ETL engine for Node.js and TypeScript with constant memory footprint.
TypeScriptMIT Licensesteady
The AI native file format. Trust scores, source provenance, and compliance metadata that embed into 20+ formats (DOCX, PDF, images, code). EXIF for AI.
PythonMIT Licenseactive
Docker microservice for saving/restoring volume data to S3.
JavaScriptdormant
A bite-sized, lightweight HDFS compatible file system built over Cassandra.
Scaladormant
Pure-Go classic machine learning toolkit and data engineering utilities. Eight algorithms with zero external dependencies.
GoMIT Licensesteady
Zero-config data quality CLI. Profiles every table on first run, then auto-detects anomalies (volume drops, schema drift, freshness misses, distribution shifts) on subsequent runs. No YAML, no rules to write. Works with Postgres, BigQuery, Snowflake, and dbt.
PythonMIT Licenseactive
A lightweight Python library for building execution pipelines with retry, parallel execution, cron scheduling, and async support.
PythonApache License 2.0active
A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via functions and processes data via columns operations as opposed to having only set operations in conventional approaches like MapReduce or SQL.
JavaMIT Licensedormant
A high-performance Python library for comparing large datasets (CSV, Parquet) locally using Rust and Polars. It features zero-copy streaming to prevent OOM errors and generates interactive HTML data quality reports.
PythonMIT Licensesteady
Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs.
PythonOtheractive
Open-source self-hosted analytics pipeline that lands raw events as Parquet in your own object storage. Uses NATS JetStream for durable buffering and BigQuery external tables for querying. Designed for teams that want to own their raw event data.
PythonApache License 2.0active
Specialized JSONL log compressor with block-level timestamp indexing and DuckDB integration. Achieves ~9% compression ratio (better than gzip) with time-range random access queries.
Otheractive
Open-source agentic data quality framework with LLM-powered diagnosis, root-cause analysis, SQL auto-fix proposals, and 31 rule types — DuckDB, Postgres, BigQuery, Databricks, Athena, Snowflake.
PythonOtheractive
Data ingestion engine that connects 400+ Singer taps to Parquet files in cloud buckets (S3, GCS, Azure). Streaming, incremental, with auto-catalog.
PythonApache License 2.0steady
Lightweight Node.js ETL framework for databases → data lakes/warehouses.
TypeScriptMIT Licensesteady
Context-Aware RAG Processing Queue for high availability and adaptive rate-limiting.
PythonMIT Licenseactive
42-table synthetic SME dataset with double-entry accounting, tax compliance (AU/US/UK), and temporal realism. CSV, SQL, Parquet, SQLite. Ideal for ETL pipeline testing.
steady
CLI tool to enrich CSV files with company data (financials, contacts, metadata) from 250M+ company records. Available on npm.
JavaScriptMIT Licenseactivearchived