Protocol Buffers - Google's data interchange format.
C++Otheractive
Awesome DataEngineering
Search 112 DataEngineering projects →112 projects16 categories
- 55%active
- 14%steady
- 10%slowing
- 21%dormant
Sourced fromigorbarinov/awesome-data-engineering
Projects 1–60 of 112, most starred first
An open-source service monitoring system and time series database.
GoApache License 2.0active
Performant open-source Python ETL framework with Rust runtime, supporting 300+ data sources.
PythonOtheractive
The easy, open source way for everyone in your company to ask questions and learn from data.
ClojureOtheractive
A system to programmatically author, schedule, and monitor data pipelines.
PythonApache License 2.0active
A distributed NewSQL database compatible with MySQL protocol.
GoApache License 2.0active
Scalable datastore for metrics, events, and real-time analytics.
RustApache License 2.0active
Scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
JavaApache License 2.0active
Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python.
PythonMIT Licenseactive
Analyzes resource usage and performance characteristics of running containers.
GoOtheractive
A Python module that helps you build complex pipelines of batch jobs.
PythonApache License 2.0active
Replicated SQLite using the Raft consensus protocol.
GoMIT Licenseactive
A cluster manager, designed for both long-lived services and short-lived batch processing workloads.
GoOtheractive
An open-source Python library for building data applications.
PythonApache License 2.0active
An embedded vector database for on-device RAG and edge AI, the SQLite of vector databases.
C++Apache License 2.0active
An open-source graph database. Google.
GoApache License 2.0active
A high-performance Cloud-Native file system driven by object storage for large-scale data storage.
GoApache License 2.0active
An open source ETL framework to build fresh index for AI.
RustApache License 2.0active
Open-source platform for data preparation, synthetic data generation, and AI/data pipelines. Includes reusable skills for automating workflow steps across data and AI tasks.
PythonApache License 2.0active
Kafka in Docker.
ShellApache License 2.0slowing
Weaving Docker containers into applications.
GoApache License 2.0slowingarchived
A fast compressor/decompressor. Used with Parquet.
C++Otheractive
A fast and efficient object graph serialization framework for Java.
HTMLBSD 3-Clause "New" or "Revised" Licenseactive
An open source platform that delivers resilience and manageability to object-storage based data lakes.
GoApache License 2.0active
A scalable, distributed Time Series Database.
JavaGNU Lesser General Public License v2.1slowing
PostgreSQL Extension that allows creating an index backed by Elasticsearch.
PLpgSQLOtherslowingarchived
A warehouse-first Customer Data Platform that enables you to collect data from every application, website and SaaS platform, and then activate it in your warehouse and business tools.
GoOtheractive
CLI tool to copy data between databases with a single command. Supports 50+ sources including PostgreSQL, MySQL, MongoDB, Salesforce, Shopify to any data warehouse.
GoOtheractive
An in-memory database and application server.
LuaOtheractive
Data Acquisition and Processing Made Easy. Deprecated.
GoOtherslowingarchived
Easily manage Docker containers & their data.
PythonApache License 2.0dormant
A distributed, fault-tolerant graph database by Twitter. Deprecated.
ScalaOtherdormantarchived
An open-source, unified metadata management for data lakes, data warehouses, and external catalogs.
JavaApache License 2.0active
JDBC importer for Elasticsearch.
JavaApache License 2.0activearchived
The Streaming SQL Database.
CApache License 2.0dormant
Node.js client for Apache Kafka 0.8.
JavaScriptMIT Licenseslowing
Python helpers for building dashboards using Flask and React.
JavaScriptMIT Licenseactivearchived
Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing.
PythonOtheractive
Pinterest's Kafka to S3 distributed consumer.
JavaApache License 2.0steady
A large-scale graph database.
JavaApache License 2.0slowingarchived
Fast scalable time series database.
JavaApache License 2.0steady
Python interface to Hive and Presto.
PythonOthersteady
The open-source reverse ETL, data activation platform for modern data teams.
RubyGNU Affero General Public License v3.0active
End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.
GoApache License 2.0active
A Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables.
JavaApache License 2.0active
Distributed. Columnar. Versioned. Streaming. SQL.
ScalaApache License 2.0active
A scalable, searchable key-value store. Deprecated.
C++BSD 3-Clause "New" or "Revised" Licenseslowing
A file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack.
PythonOtheractive
An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT, and the native Kafka protocol.
JavaOtheractive
Typed graph database where agents branch and merge like Git. S3-native, Rust, traversal + vector + BM25 in one runtime.
RustMIT Licenseactive
DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs.
JavaScriptApache License 2.0dormantarchived
Pinterest MySQL Management Tools.
PythonGNU General Public License v2.0dormantarchived
A pure python HDFS client.
PythonApache License 2.0dormantarchived
A scalable time series database based on Cassandra and Elasticsearch, by Spotify.
JavaApache License 2.0dormantarchived
A numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
C++Apache License 2.0dormantarchived
The streaming database built for IoT data storage and real-time processing.
HaskellBSD 3-Clause "New" or "Revised" Licenseslowing
Fast distributed metrics database.
ErlangMIT Licensedormant
Package golang service into minimal Docker containers.
ShellBSD 3-Clause "New" or "Revised" Licensedormant
A Python library that facilitates the comparison of two DataFrames in Pandas, Polars, Spark and more. The library goes beyond basic equality checks by providing detailed insights into discrepancies at both row and column levels.
PythonApache License 2.0active
Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption.
GoApache License 2.0dormantarchived