DataEngineering · Data Ingestion

Data Ingestion

• wurstmeister/kafka-docker on GitHub, opens in a new tab

  7Kstars

  Kafka in Docker.

  Shell·Apache License 2.0·slowing·2 years ago

• bruin-data/ingestr on GitHub, opens in a new tab

  3.8Kstars

  CLI tool to copy data between databases with a single command. Supports 50+ sources including PostgreSQL, MySQL, MongoDB, Salesforce, Shopify to any data warehouse.

  Go·Other·active·9 hours ago

• mozilla-services/heka on GitHub, opens in a new tab

  3.4Kstars

  Data Acquisition and Processing Made Easy. Deprecated.

  Go·Other·slowing·3 years agoarchived

• SOHU-Co/kafka-node on GitHub, opens in a new tab

  2.7Kstars

  Node.js client for Apache Kafka 0.8.

  JavaScript·MIT License·slowing·3 years ago

• pinterest/secor on GitHub, opens in a new tab

  1.9Kstars

  Pinterest's Kafka to S3 distributed consumer.

  Java·Apache License 2.0·steady·5 months ago

• airbnb/kafkat on GitHub, opens in a new tab

  502stars

  Simplified command-line administration for Kafka brokers.

  Ruby·Apache License 2.0·dormant·7 years ago

• kroxylicious/kroxylicious on GitHub, opens in a new tab

  308stars

  A Kafka Proxy, solving problems like encrypting your Kafka data at rest.

  Java·Apache License 2.0·active·6 hours ago

• xstevens/pg_kafka on GitHub, opens in a new tab

  112stars

  A PostgreSQL extension to produce messages to Apache Kafka.

  C·MIT License·dormant·11 years agoarchived

• NameetP/pdfmux on GitHub, opens in a new tab

  79stars

  Python PDF-to-Markdown orchestrator. Classifies each page and routes to the optimal backend (PyMuPDF, Docling, RapidOCR, Gemini Flash), emitting Markdown plus a per-page confidence score so ingestion pipelines can quarantine low-trust pages before feeding LLMs or retrieval.

  Python·MIT License·active·4 days ago

• uber/kafka-logger on GitHub, opens in a new tab

  45stars

  Kafka-winston logger for Node.js from Uber.

  JavaScript·MIT License·dormant·8 years agoarchived

• drt-hub/drt on GitHub, opens in a new tab

  29stars

  OSS Reverse ETL CLI. Sync data from warehouses to business tools via YAML.

  Python·Apache License 2.0·active·15 hours ago

• fulldecent/google-sheets-etl on GitHub, opens in a new tab

  22stars

  Live import all your Google Sheets to your data warehouse.

  PHP·MIT License·steady·6 months ago

• pujansrt/data-genie on GitHub, opens in a new tab

  16stars

  High-performance, streaming-first ETL engine for Node.js and TypeScript with constant memory footprint.

  TypeScript·MIT License·steady·3 months ago

• mgillr/crdt-merge on GitHub, opens in a new tab

  6stars

  Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs.

  Python·Other·active·last week

• mirlan-irokez/rawbbit on GitHub, opens in a new tab

  6stars

  Open-source self-hosted analytics pipeline that lands raw events as Parquet in your own object storage. Uses NATS JetStream for durable buffering and BigQuery external tables for querying. Designed for teams that want to own their raw event data.

  Python·Apache License 2.0·active·23 hours ago

• dataspoclab/dataspoc-pipe on GitHub, opens in a new tab

  2stars

  Data ingestion engine that connects 400+ Singer taps to Parquet files in cloud buckets (S3, GCS, Azure). Streaming, incremental, with auto-catalog.

  Python·Apache License 2.0·steady·4 months ago

• bahador-r/db2lake on GitHub, opens in a new tab

  2stars

  Lightweight Node.js ETL framework for databases → data lakes/warehouses.

  TypeScript·MIT License·steady·11 months ago

• whispering3/CARQ on GitHub, opens in a new tab

  2stars

  Context-Aware RAG Processing Queue for high availability and adaptive rate-limiting.

  Python·MIT License·active·2 months ago

• Alessandro114/enrich-companies on GitHub, opens in a new tab

  0stars

  CLI tool to enrich CSV files with company data (financials, contacts, metadata) from 250M+ company records. Available on npm.

  JavaScript·MIT License·active·2 months agoarchived